[asio-users] Problem with boost::asio::ssl when using multiple threads

[Christian Hägele]

Hello,

I have a problem when using boost::asio::ssl with multiple-threads. My
application runs fine on a lot of servers without any problems when not
using SSL. So I suspect the code is pretty much thread-safe. All
I/O-Operations are guarded by strands.
But when using boost::asio::ssl with OpenSSL I run into occasional
crashes. I think I tracked down the problem to the following fact:

In boost/asio/ssl/detail/io.hpp the function operator() is called by 2
different threads at the same time. One thread is going through a explicit
strand in my user-code for that connection, the other one is the result of
a previous async_write or async_read started from that
io_op::operator()-function. That happens quite a lot. (Remark there is
only 1 connection during testing!)
Because of that boost::asio::ssl::detail::engine::perform is called by 2
different threads concurrently. I am not that familiar with OpenSSL, but I
suppose that this could be problematic.

Do I miss anything in my observation? Does somebody else has
boost::asio::ssl working in a multi threaded environment (meaning multiple
threads and one io_service) without problems? Does anybody has enough
knowledge of OpenSSL to say if that access could be problematic?

Reagrds,

Christian

[Александр Бобров]

You can read here about openssl (the same issue):

http://stackoverflow.com/questions/14467630/clarifying-openssl-0-9-8l-concurrency-support-can-ssl-instances-be-used-by-mul

It seems that you run concurrently async_read / async_write on the same
boost::asio::ssl::stream object (full duplex), so SSL_read and SSL_write
can run concurrently.

Strands is only for IO operations completition handlers. async_read and
async_write are composed from other asynchronous operations, and all
intermediate handlers should be invoked using the same method as the final
handler.

You should implement your own asio_handler_invoke like it's done in asio
docs:

http://www.boost.org/doc/libs/1_53_0/doc/html/boost_asio/reference/asio_handler_invoke.html

Post by Christian Hägele
Hello,
I have a problem when using boost::asio::ssl with multiple-threads. My
application runs fine on a lot of servers without any problems when not
using SSL. So I suspect the code is pretty much thread-safe. All
I/O-Operations are guarded by strands.
But when using boost::asio::ssl with OpenSSL I run into occasional
In boost/asio/ssl/detail/io.hpp the function operator() is called by 2
different threads at the same time. One thread is going through a explicit
strand in my user-code for that connection, the other one is the result of
a previous async_write or async_read started from that
io_op::operator()-function. That happens quite a lot. (Remark there is
only 1 connection during testing!)
Because of that boost::asio::ssl::detail::engine::perform is called by 2
different threads concurrently. I am not that familiar with OpenSSL, but I
suppose that this could be problematic.
Do I miss anything in my observation? Does somebody else has
boost::asio::ssl working in a multi threaded environment (meaning multiple
threads and one io_service) without problems? Does anybody has enough
knowledge of OpenSSL to say if that access could be problematic?
Reagrds,
Christian
------------------------------------------------------------------------------
Free Next-Gen Firewall Hardware Offer
Buy your Sophos next-gen firewall before the end March 2013
and get the hardware for free! Learn more.
http://p.sf.net/sfu/sophos-d2d-feb
_______________________________________________
asio-users mailing list
https://lists.sourceforge.net/lists/listinfo/asio-users
_______________________________________________
Using Asio? List your project at
http://think-async.com/Asio/WhoIsUsingAsio

[Christian Hägele]

Am 05.02.2013, 19:42 Uhr, schrieb Александр Бобров

Post by Александр Бобров
http://stackoverflow.com/questions/14467630/clarifying-openssl-0-9-8l-concurrency-support-can-ssl-instances-be-used-by-mul
It seems that you run concurrently async_read / async_write on the same
boost::asio::ssl::stream object (full duplex), so SSL_read and SSL_write
can run concurrently.

That Stackoverflow article is quite interesting.
It seems that the boost::asio::ssl can't be used in a multithread
environment unless you take special care not to async_send and
async_receive at the same on the boost::asio::ssl_stream.
It took me quite a long time to figure it out. The documentation could be
better there.

Post by Александр Бобров
Strands is only for IO operations completition handlers. async_read and
async_write are composed from other asynchronous operations, and all
intermediate handlers should be invoked using the same method as the final
handler.
You should implement your own asio_handler_invoke like it's done in asio
http://www.boost.org/doc/libs/1_53_0/doc/html/boost_asio/reference/asio_handler_invoke.html

I don't understand how I could do this properly. Because I have one strand
per object how can I set a different handler-invoker-hook for every
async-operation?

Actually I am thinking of a workaround to protect the OpenSSL-Object by
inserting a mutex in boost::asio::ssl::detail::io_op::operator(). I have
to test the code more, though.


Regards,

Christian

[Christian Hägele]

Am 05.02.2013, 19:42 Uhr, schrieb Александр Бобров

Post by Александр Бобров
It seems that you run concurrently async_read / async_write on the same
boost::asio::ssl::stream object (full duplex), so SSL_read and SSL_write
can run concurrently.
Strands is only for IO operations completition handlers. async_read and
async_write are composed from other asynchronous operations, and all
intermediate handlers should be invoked using the same method as the final
handler.

I finally found my problem after weeks of investigation!
I made the following mistake:
I used a helper function which looks as follows:

void SendCompleteQueueInternal(const
std::vector<boost::asio::const_buffer> &cb,
const boost::function<void (const boost::system::error_code &e,
std::size_t bytes_transferred)> &callback)
{
m_SslSocket.async_write_some(cb, callback);
}

That function was called like this:

SendCompleteQueueInternal(bufs, m_Strand.wrap(boost::bind(&HandleWrite,
m_This.lock(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred)));

So I used a strand to protect concurrent access, but because my
Helper-Function used a boost::function-object as handler instead of the
type that is returned by m_Strand.wrap boost::asio's integrated
specialization for wrapped_handlers didn't work and that's why there were
crashes and heap-corruption.
I used a similar Helper-Function on regular TCP-Connections, too, but the
issue did not show up there, because I didn't use any composed operations
(just async_read_some and async_write_some).

So it was a mistake by myself and no issue in the
boost::asio::ssl-implementation.

Regards,

Christian

[vf]

The ssl thing normally works with boost asio alright.
How does exactly it crash, when reading, writing or disconnecting?
How did you arrive at your observation?
What boost version is involved?

[Christian Hägele]

Am 07.02.2013, 06:54 Uhr, schrieb vf

Post by vf
The ssl thing normally works with boost asio alright.
How does exactly it crash, when reading, writing or disconnecting?
How did you arrive at your observation?
What boost version is involved?

There where several problem I encountered:

1. The async_shutdown-method does not fit in the rest of our design. That
problem was our fault. But the async_shutdown is hard to implement right,
because you have to cancel all your outstanding async-operations (if I'm
right in that case?).

2. The SSL-connection is running fine for hour, but from time to time the
read-functino failed with a SSL-error-code (decryption failed) that should
not happen. I believe that data got mixed up somewhere.

3. I think as a result of 2) heap corruption happened. Sometimes a crash
in a memcpy-operation in SSL-BIO-Interface.

After a LOT of tests and tries I figured out to reproduce 2). It fairly
easy to reproduce when I sent a lot of data in both directions, but only
happens when multiple worker-threads are running.

Regards,

Christian

[vf]

The reason I was asking is that typical problems arise from either
(a) improper disconnection and clean up, i.e. when a handler uses
some data and is invoked after this data is cleaned up.
(b) when a composed operation such as async_write() is performed in
sequence, i.e
async_write( some_data_to_send )
async_write( some_more_data_to_send )

...

async_read()

In the latter case the data from the second async_write() may interleave with
the data from the previous async_write().

Though it might seem to work with the non-ssl code, the ssl version is generally
much slower due to encrypting and the size of the actual data sent may be
noticeably bigger. Therefore these kind of problems are more likely to surface
in the ssl version.