Synapse send stream issue

Discussion and questions about programming with Ultibo.
sbk58
Posts: 14
Joined: Fri Jul 21, 2017 7:48 pm

Synapse send stream issue

Postby sbk58 » Tue Jun 12, 2018 1:53 pm

Hi;

First thanks for the fix to the built in uart driver works great no more dropped characters.

I have been using the Utilbo version of the synapse socket library to port a number of projects to Ultibo. After the May 1st Ultibo update the SendStreamRaw method will not send more then @ 310K of a source stream.

After about 310K the synapse socket reports an error 10053. I do not know what a 10053 means in Ultibo but in Windows it indicates a low level protocol error. The source stream being sent is always a file stream (TFileStream). The error occurs on all of my ethernet equipped raspi devices (B+, 2, 3B). The socket used for the transfer is set to use blocking sockets. It does not matter whether the socket is connected by calling connect or using the accept method of a listen socket.

I ported my project to a pi3b running the latest raspbian linux to allow the use of the debugger in Lazarus. I used the Ultibo version of the Synapse library. Of course the project works perfectly under raspbian so I was not able to locate the issue.

Anyone else seen this issue?

Thanks...
User avatar
Ultibo
Site Admin
Posts: 1979
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: Synapse send stream issue

Postby Ultibo » Tue Jun 12, 2018 11:30 pm

sbk58 wrote:I have been using the Utilbo version of the synapse socket library to port a number of projects to Ultibo. After the May 1st Ultibo update the SendStreamRaw method will not send more then @ 310K of a source stream.

After about 310K the synapse socket reports an error 10053. I do not know what a 10053 means in Ultibo but in Windows it indicates a low level protocol error. The source stream being sent is always a file stream (TFileStream). The error occurs on all of my ethernet equipped raspi devices (B+, 2, 3B). The socket used for the transfer is set to use blocking sockets. It does not matter whether the socket is connected by calling connect or using the accept method of a listen socket.

Hi,

I'll try to create a simple test program to see if I get the same results (unless you already have a simplified test case you can post), the 10053 error means WSAECONNABORTED (Connection Aborted) which is the same meaning as in Windows.

Do you have something like Wireshark available to capture the conversation so we can see what is happening between the two ends?

Is the other end of the connection Linux or Windows (or does it happen for either)?

Thanks.
Ultibo.org | Make something amazing
https://ultibo.org
sbk58
Posts: 14
Joined: Fri Jul 21, 2017 7:48 pm

Re: Synapse send stream issue

Postby sbk58 » Wed Jun 13, 2018 2:14 am

Thanks for the quick response;

I have carved out the ftp server engine from the project in to a separate test program. I think it will be simpler then attempting to extract the http server engine. I have attached a zip file with the test program and all of the required units, let me know if I missed any. You will need to change the path for the synapse library. The transfer is performed by the "ftptransferthreads" unit.

The problem is both with windows and linux clients. If you would like I can do a capture using wireshark on my Ubuntu system. I think the example might be better as it allows you to do your own capture.
Attachments
tinyftp.zip
(64.19 KiB) Downloaded 13 times
User avatar
Ultibo
Site Admin
Posts: 1979
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: Synapse send stream issue

Postby Ultibo » Wed Jun 13, 2018 11:06 am

sbk58 wrote:I have carved out the ftp server engine from the project in to a separate test program.

Thanks for that, I am able to compile the test and it does reproduce the failure exactly as you describe.

At first look it appears that nothing abnormal shows in a Wireshark capture so the problem seems to be internal to the TCP protocol layer.

Give us a few days to look into it and see what we can find.
Ultibo.org | Make something amazing
https://ultibo.org
User avatar
Ultibo
Site Admin
Posts: 1979
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: Synapse send stream issue

Postby Ultibo » Fri Jun 15, 2018 3:41 am

We've just committed a fix for our port of Synapse which resolves this issue in our testing.

The problem related to Synapse passing a TTimeval to the SetSockOpt function when setting the send and receive timeout values, under Winsock (which is what the Ultibo sockets API provides) the value passed is just a longint. With these changes in place your test application functions perfectly for all transfer sizes.

Please download the latest copy of Synapse from our GitHub link above and let us know your result.

Thanks again for providing the test application which made this very easy to diagnose.
Ultibo.org | Make something amazing
https://ultibo.org
sbk58
Posts: 14
Joined: Fri Jul 21, 2017 7:48 pm

Re: Synapse send stream issue

Postby sbk58 » Sat Jun 16, 2018 2:20 am

Thanks for the quick fix to the synapse library.

The fix seams to solve the SendStreamRaw issue. Once I remembered that I set the synapse transfer buffer size to 16K and set it back to 64K transfer speed is now twice what I used to see before the May 1st and 7th updates.

I also used to see a random delay connecting a client to the ftp server control channel (could take up to 6 sec) that also seams to be fixed.

When using FileZilla client set to passive mode small file transfers to the ftp server fail to connect to the server. When I use WireShark to watch the transfer it appears the data is sent from the client to the server and the server ACKs the data but the server code can not complete the connection (accept fails). Before the May updates this happened randomly for small files (< 1.5 K) uploaded to the server in passive mode. Since the May updates it now happens all the time for files < 100K. I do not see this with any other client software.

Some testing shows that FileZilla does not wait for an intermediate response from the server when uploading a file in passive mode and begins the transfer immediately after sending the ftp STOR command. When I compile my test server for linux or windows the transfers always complete. I can not understand how the data is ACKed by the server but accept fails. Seams like the client connection completes before I call accept on the connection and the data is lost. Any idea why this may be happening.
User avatar
Ultibo
Site Admin
Posts: 1979
Joined: Sat Dec 19, 2015 3:49 am
Location: Australia

Re: Synapse send stream issue

Postby Ultibo » Mon Jun 18, 2018 5:35 am

sbk58 wrote:Some testing shows that FileZilla does not wait for an intermediate response from the server when uploading a file in passive mode and begins the transfer immediately after sending the ftp STOR command. When I compile my test server for linux or windows the transfers always complete. I can not understand how the data is ACKed by the server but accept fails. Seams like the client connection completes before I call accept on the connection and the data is lost. Any idea why this may be happening.

This turned out to be an interesting case of timing, when the file was small enough it could be transferred completely and FileZilla would close the connection (sending a FIN to the server) before your receiving thread called accept in the listening socket. Once the FIN had been received the socket state moved to Disconnecting and the Accept function would not select it from the queue anymore.

We've just committed a very small change to the TCP unit which causes Accept to also select sockets that have moved to Disconnecting state which resolves this case.

If you download and apply the latest RTL (and rebuild the RTL as per the Wiki instructions) the problem should be gone, please let us know your results.
Ultibo.org | Make something amazing
https://ultibo.org
sbk58
Posts: 14
Joined: Fri Jul 21, 2017 7:48 pm

Re: Synapse send stream issue

Postby sbk58 » Wed Jun 20, 2018 10:38 am

Good catch;

Just tested the updates you posted and they seam to have fixed the problem. I have tested this on my pi2 setup will let you know if there are any issue once I get a chance to test on all raspi device types.

While it is more efficient to use the Ultibo tcp code directly the synapse library allows an application to be developed that can be compiled on lazarus under raspbian allowing for full debugging in the lazarus IDE.

Thanks...

Return to “General”

Who is online

Users browsing this forum: No registered users and 0 guests