[asio-users] Composed operations not always atomic with strands

[Bjorn Reese]

I am using async_write() together with a strand in a multi-threaded
program in the same way as in the HTTP server 3 example.

Under heavy load the async_write() operation ceases to be atomic (i.e.
data send by one operation may be interleaved by data from another.)

I have created the attached test case to illustrate the problem. Here
I have two threads sending blocks of data on the same socket. The
receiver is single-threaded, but I have added a sleep() to simulate
slow processing.

When the TCP buffer on the receiver side is full, the sender will only
send part of the data. While the rest will be sent eventually, the
problem is that data submitted by the other thread may be sent
inbetween, so the data is received in the wrong order. The receiver
will assert if this is the case.

Before I create a ticket for this, can anybody find any errors in my
attached examples?

[Dale Wilson]

Post by Bjorn Reese
I am using async_write() together with a strand in a multi-threaded
program in the same way as in the HTTP server 3 example.
Under heavy load the async_write() operation ceases to be atomic (i.e.
data send by one operation may be interleaved by data from another.)
I have created the attached test case to illustrate the problem. Here
I have two threads sending blocks of data on the same socket. The
receiver is single-threaded, but I have added a sleep() to simulate
slow processing.
When the TCP buffer on the receiver side is full, the sender will only
send part of the data. While the rest will be sent eventually, the
problem is that data submitted by the other thread may be sent
inbetween, so the data is received in the wrong order. The receiver
will assert if this is the case.
Before I create a ticket for this, can anybody find any errors in my
attached examples?

Strand was not designed to provide the type of thread-safety you are
looking for. It only ensures that only one handler/callback method can
execute at any time.

It does not magically (or otherwise) convert asynchronous operations
into atomic ones. You might solve your problem by switching to
synchronous writes rather than using async_write()

Dale

Dale

Post by Bjorn Reese
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
_______________________________________________
asio-users mailing list
https://lists.sourceforge.net/lists/listinfo/asio-users
_______________________________________________
Using Asio? List your project at
http://think-async.com/Asio/WhoIsUsingAsio

[Bjorn Reese]

Post by Dale Wilson
Strand was not designed to provide the type of thread-safety you are
looking for. It only ensures that only one handler/callback method can
execute at any time.

I disagree, but let us leave multi-threading out of the picture, because
that is not the problem here.

I have modified the sender to be single-threaded. Now it performs two
sequential async_write() operations per callback. It exposes the exact
same behavior as I have described before, which is...

Normally it will send the entire A package, and then the entire B
package. However, if the receiver buffer is full, then it will send
half of the A package, then the entire B package, and finally the
rest of the A package.

To provide some context, I use this in a messaging middleware (currently
based on the ZeroMQ protocol), where each package contains a length
followed by a payload. The abovementioned behavior means that the
middleware will fail under heavy load. There is no way that the receiver
can compensate for the strange behavior of the sender.

[Bjorn Reese]

Post by Bjorn Reese
Normally it will send the entire A package, and then the entire B
package. However, if the receiver buffer is full, then it will send
half of the A package, then the entire B package, and finally the
rest of the A package.

As follow-up, I have located the problem the the epoll reactor. If I
compile with BOOST_ASIO_DISABLE_EPOLL, then the problem disappears.

[svante karlsson]

I think Dale Wilson is right. Strands gives a guarantee of not invoking
multiple io ops simultaneously. This promise still holds. You should
complete the send of one message before starting a new one.

/svante

Post by Bjorn Reese

Post by Bjorn Reese
Normally it will send the entire A package, and then the entire B
package. However, if the receiver buffer is full, then it will send
half of the A package, then the entire B package, and finally the
rest of the A package.

As follow-up, I have located the problem the the epoll reactor. If I
compile with BOOST_ASIO_DISABLE_EPOLL, then the problem disappears.
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
_______________________________________________
asio-users mailing list
https://lists.sourceforge.net/lists/listinfo/asio-users
_______________________________________________
Using Asio? List your project at
http://think-async.com/Asio/WhoIsUsingAsio

[David Schwartz]

Post by Dale Wilson
Strand was not designed to provide the type of thread-safety you are

Post by Dale Wilson
looking for. It only ensures that only one handler/callback method can
execute at any time.

I disagree, but let us leave multi-threading out of the picture, because
that is not the problem here.
I have modified the sender to be single-threaded. Now it performs two
sequential async_write() operations per callback. It exposes the exact
same behavior as I have described before, which is...
Normally it will send the entire A package, and then the entire B
package. However, if the receiver buffer is full, then it will send
half of the A package, then the entire B package, and finally the
rest of the A package.

It is doing what you are asking it to do. You are starting one operation
before the other finishes, so one operation starts before the other
finishes. If you don't want an operation to start until the previous
operation finishes, don't start an operation until the previous operation
finishes. Code what you want.

DS

[Dale Wilson]

Post by Bjorn Reese

Post by Dale Wilson
Strand was not designed to provide the type of thread-safety you are
looking for. It only ensures that only one handler/callback method can
execute at any time.

I disagree, but let us leave multi-threading out of the picture, because
that is not the problem here.

Just for the record, if you are doing async operations you cannot leave
multithreading out of the picture. But I agree multithreading is not
the problem.

Post by Bjorn Reese
I have modified the sender to be single-threaded. Now it performs two
sequential async_write() operations per callback. It exposes the exact
same behavior as I have described before, which is...

So in other words you have given ASIO a second asynchronous write
request while the first one was still active. Because, as you have
noted, asynchronous write is not atomic (and because strand does not
magically make it atomic) this is invalid behavior from your application
and the results will be indeterminate.

Post by Bjorn Reese
Normally it will send the entire A package, and then the entire B
package.

Let me rephrase that. "Accidentally if everything goes well it will
send the entire contents of A package then the entire B package."
This behavior is not guaranteed, nor should it be.

Post by Bjorn Reese
However, if the receiver buffer is full, then it will send
half of the A package, then the entire B package, and finally the
rest of the A package.
To provide some context, I use this in a messaging middleware (currently
based on the ZeroMQ protocol), where each package contains a length
followed by a payload. The abovementioned behavior means that the
middleware will fail under heavy load. There is no way that the receiver
can compensate for the strange behavior of the sender.

I understand the problem. In fact I just recently solved a similar
one. However my solution was -- and your solution must be --in the
application code, not in ASIO.

async_write is composed on top of async_write_some. Most of the time
async_write some will write all of the data requested. That's the
"accidentally correct" behavior you are seeing. However, sometimes it
won't. When it doesn't it schedules another request to finish writing
the "package". At this point you have two requests in ASIO for the
same socket. One will write some or all of package B, the other will
write the second portion of package A. How is ASIO supposed to know
which one it should start next?

It could do FIFO --- but that would fail because the Package B request
was placed before the second Package A request was placed. Or it could
have an elaborate mechanism to mark the socket busy servicing the
package A request(s) and have the Package B (and C and D and E?)
requests wait patiently for their turn. That mechanism does not
presently exist in ASIO, and it would be a major redesign to add it.

Request scheduling is certainly not the intent of strand. Strand lets
you write "single threaded" code in your handlers. It has no impact on
choosing which of multiple outstanding requests for the same socket
should be started next. Hoping that strand will solve the problem for
you will not work (sorry) -- it's the wrong tool for the job. Strand
solves multithreading issues, and as you have pointed out, this is not a
multithreading problem.

So you have two choices -- switch to synchronous blocking requests. Your
single threaded code will block until package A is completely delivered
before attempting to send package B. Or provide your own queuing
mechanism so package B is not presented to ASIO until package A has been
completely sent. Either of these will work. Which one you choose
depends on a trade-off between throughput needs and code complexity.
Blocking is (much!) simpler, but it may be slower overall.

Dale

Post by Bjorn Reese
------------------------------------------------------------------------------
Start Your Social Network Today - Download eXo Platform
Build your Enterprise Intranet with eXo Platform Software
Java Based Open Source Intranet - Social, Extensible, Cloud Ready
Get Started Now And Turn Your Intranet Into A Collaboration Platform
http://p.sf.net/sfu/ExoPlatform
_______________________________________________
asio-users mailing list
https://lists.sourceforge.net/lists/listinfo/asio-users
_______________________________________________
Using Asio? List your project at
http://think-async.com/Asio/WhoIsUsingAsio

[Bjorn Reese]

Post by Dale Wilson
So in other words you have given ASIO a second asynchronous write
request while the first one was still active. Because, as you have
noted, asynchronous write is not atomic (and because strand does not
magically make it atomic) this is invalid behavior from your application
and the results will be indeterminate.

Having reviewed the documentation and the code, I agree with you that
strands do not provide the required guarantees.

I ended up creating a socket wrapper that contains an async_write()
member function which uses an internal (lock-free) message queue to
make the transmission atomic.

In case somebody is looking for a similiar solution, the code can be
found here:

http://sourceforge.net/p/axiomq/code/ci/master/tree/include/axiomq/basic_queue_socket.hpp

[Roger Austin (Australia)]

This issue is probably the most-frequently-asked question on the asio mailing lists. It seems to pop up every 6 months or so. As others have said:

This behaviour is by design.
You cannot safely call async_write on a socket until the previous async_write has completed.
The usual solution is to provide your own message queue (thread-safe, if you are multi-threading). See the asio chat sample for an example.


-----Original Message-----
From: Bjorn Reese [mailto:***@mail1.stofanet.dk]
Sent: Thursday, 24 April 2014 6:45 PM
To: asio-***@lists.sourceforge.net
Subject: Re: [asio-users] Composed operations not always atomic with strands

Post by Dale Wilson
Strand was not designed to provide the type of thread-safety you are
looking for. It only ensures that only one handler/callback method
can execute at any time.

I disagree, but let us leave multi-threading out of the picture, because that is not the problem here.

I have modified the sender to be single-threaded. Now it performs two sequential async_write() operations per callback. It exposes the exact same behavior as I have described before, which is...

Normally it will send the entire A package, and then the entire B package. However, if the receiver buffer is full, then it will send half of the A package, then the entire B package, and finally the rest of the A package.

To provide some context, I use this in a messaging middleware (currently based on the ZeroMQ protocol), where each package contains a length followed by a payload. The abovementioned behavior means that the middleware will fail under heavy load. There is no way that the receiver can compensate for the strange behavior of the sender.