[asio-users] semantics of write callbacks

[Adam Crain]

Hi,

I have a question about the semantics of "async_write" and the callback.
We are observing unexpected TCP fragmentation that I would think Nagle's
algortihm would optimize for us...

In the protocol we implement, frames get packet-ized into chunks no bigger
than 250 bytes. In some systems we do an application layer ACK for every
frame b/c the protocol was originally design for noisy RS232 serial comms
(it has it's own CRC and transport reassembly).

On TCP, however, the data fragments can be sent back-to-back with no
application ACKs... I would expect that Nagle's in the TCP stack would turn
a series of small async_write calls into a larger TCP frame, but that
doesn't appear to be happening.

What does the callback from async_write mean? Does it mean that the data is
queued for transmission on the stream or does it mean "I packetized this
for you" and shipped it out.

We are definitely NOT disabling Nagle's via the TCP_NODELAY option.

Pardon my ignorance of how this stuff works! The manual for ASIO is quite
good, but I didn't know how to interpret it on this issue.

thank you,
Adam
--
J Adam Crain

Owner - Automatak <http://www.automatak.com>, LLC

pub 4096R/E2984A0C
<http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x9B78643BE2984A0C>
2013-05-03 John Adam Crain (Business PGP key)
<***@automatak.com>
<http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x9B78643BE2984A0C>

[Roger Austin (Australia)]

Hi Adam,

asio provides a fairly thin layer over the TCP stack. Each call to async_write will put at least one packet on the stack (it could put more, via successive calls to async_write_some, if the buffer passed to async_write contains more data than the TCP stack will accept in one operation). The callback from async_write occurs when the last data in the buffer has been handed off to the TCP stack (i.e. 'queued for transmission' rather than 'shipped out'). I don't believe that asio should interfere with Nagling in the TCP stack although it is conceivable (but unlikely) that the asio implementation tweaks the socket options.

Don't forget that the Nagle algorithm only combines packets if it is still waiting for a TCP ACK from a sent packet. If the packets are getting acknowledged fast enough then they will not be combined.

HTH

Roger

From: Adam Crain [mailto:***@automatak.com]
Sent: Friday, 28 June 2013 07:48
To: asio-***@lists.sourceforge.net
Subject: [asio-users] semantics of write callbacks

Hi,

I have a question about the semantics of "async_write" and the callback. We are observing unexpected TCP fragmentation that I would think Nagle's algortihm would optimize for us...

In the protocol we implement, frames get packet-ized into chunks no bigger than 250 bytes. In some systems we do an application layer ACK for every frame b/c the protocol was originally design for noisy RS232 serial comms (it has it's own CRC and transport reassembly).

On TCP, however, the data fragments can be sent back-to-back with no application ACKs... I would expect that Nagle's in the TCP stack would turn a series of small async_write calls into a larger TCP frame, but that doesn't appear to be happening.

What does the callback from async_write mean? Does it mean that the data is queued for transmission on the stream or does it mean "I packetized this for you" and shipped it out.

We are definitely NOT disabling Nagle's via the TCP_NODELAY option.

Pardon my ignorance of how this stuff works! The manual for ASIO is quite good, but I didn't know how to interpret it on this issue.

thank you,
Adam

--

J Adam Crain

Owner - Automatak<http://www.automatak.com>, LLC

pub 4096R/E2984A0C<http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x9B78643BE2984A0C> 2013-05-03 John Adam Crain (Business PGP key) <***@automatak.com><http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x9B78643BE2984A0C>

[Marat Abrarov]

Post by Roger Austin (Australia)
The callback from async_write occurs when the last data
in the buffer has been handed off to the TCP stack
(i.e. 'queued for transmission' rather than 'shipped out').

It's just OS/TCP stack dependent. If you turn off send buffer of socket (set
its size to 0) then handler will be called when data is 'shipped out'. I
don't think you (your code) have to be familiar with this.

Post by Roger Austin (Australia)
Don't forget that the Nagle algorithm only combines packets
if it is still waiting for a TCP ACK from a sent packet.
If the packets are getting acknowledged fast enough then they will not be

combined.

Also note that Nagle's algorithm just "defers" sending of packets for some
timeout. It won't wait for TCP ACK for too long even if outgoing packet is
small.

Regards,
Marat Abrarov.

[Marat Abrarov]

I think boost::asio::buffered_write_stream
(http://www.boost.org/doc/libs/1_53_0/doc/html/boost_asio/reference/buffered
_write_stream.html) can help you.

Create it with large enough buffer size and use
buffered_write_stream::async_write_some. Before closing the socket use
buffered_write_stream::(async_)flush.

Regards,
Marat Abrarov.

[Marat Abrarov]

I think boost::asio::buffered_write_stream can help you.
Create it with large enough buffer size and use
buffered_write_stream::async_write_some. Before closing the socket use
buffered_write_stream::(async_)flush.

Or use asio::async_write(_xxxx) with buffered_write_stream because
asio::buffered_write_stream satisfies the requirements for AsyncWriteStream/
AsyncReadStream
(http://www.boost.org/doc/libs/1_53_0/doc/html/boost_asio/reference/AsyncWri
teStream.html).

[Adam Crain]

Roger and Marat,

Thanks for all the feedback and useful interpretation of Nagle's Alg and
options within ASIO.

I think the best solution for our project is to change some of our internal
interfaces a bit so that the entire buffer can be written to the socket
with a single async_write call despite the fact the buffer is internally
sub-divided into frames. This way the TCP stack always has the opportunity
to do the optimal thing depending on how it's configured.

It should be a pretty easy fix for us. It will use a tad bit more heap
buffer space to do all the encapsulation up front, but even my embedded
users shouldn't care about an extra 2K-3K per connection.

thanks!
Adam

Post by Marat Abrarov

I think boost::asio::buffered_write_stream can help you.
Create it with large enough buffer size and use
buffered_write_stream::async_write_some. Before closing the socket use
buffered_write_stream::(async_)flush.

Or use asio::async_write(_xxxx) with buffered_write_stream because
asio::buffered_write_stream satisfies the requirements for
AsyncWriteStream/
AsyncReadStream
(
http://www.boost.org/doc/libs/1_53_0/doc/html/boost_asio/reference/AsyncWri
teStream.html).
------------------------------------------------------------------------------
Build for Windows Store.
http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
asio-users mailing list
https://lists.sourceforge.net/lists/listinfo/asio-users
_______________________________________________
Using Asio? List your project at
http://think-async.com/Asio/WhoIsUsingAsio

--
J Adam Crain

Owner - Automatak <http://www.automatak.com>, LLC

pub 4096R/E2984A0C
<http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x9B78643BE2984A0C>
2013-05-03 John Adam Crain (Business PGP key)
<***@automatak.com>
<http://pgp.mit.edu:11371/pks/lookup?op=vindex&search=0x9B78643BE2984A0C>