[asio-users] performance problems on Linux in a multithreaded environment

[namreeb]

Hello.

A project I am working on is considering switching to ASIO to govern our
server's handling of TCP connections with clients. The server is a
multi-threaded application which exchanges a large number of small packets
with the client. The reading and writing to sockets is done on multiple
threads.

I have seen reports of performance issues with ASIO in a Linux (and
possibly, more generally, UNIX) environment relating to some internal mutex
which essentially prevents epoll() from executing across multiple threads
simultaneously.

I have started writing a small application to simulate this scenario, using
the examples in the documentation as a starting point. However, I thought
it would be prudent to also inquire to the experts about this issue here.
It seems like if this is indeed an issue, it should be straight-forward
enough for the maintainer(s) of ASIO to address?

I understand that there is a wiki article relating to this (
https://think-async.com/Asio/LinuxPerformanceImprovements) but it has not
been updated in some time.

Thank you!

[Marat Abrarov]

Hi namreeb,

some internal mutex which essentially prevents epoll() from executing across multiple threads simultaneously.

I believe, io_service-per-thread (number of threads ~ number of CPUs) should solve this. I use this way in asio_samples project (https://github.com/mabrarov/asio_samples) - the echo_server application can be configured to use:

1. single instance of boost::asio::io_service for all work (IO) threads - default configuration on Windows as Windows IOCP schedules threads better this way.
2. io_service-per-thread (number of threads == number of CPUs) - default configuration on non-Windows OS.

Unfortunately, I have no hardware machine (only virtual machine) to test performance on Linux (the second option). On Linux VM the second option shows better and more stable CPU loading then the first one.

Regards,
Marat Abrarov.

[namreeb]

Hello Marat,

Thank you for your reply. I apologize that I am not an expert on ASIO. If
I understand you correctly, using one service per thread rather than a
global service will solve (or possibly work around) the problem.

When you say that a Linux VM with this solution shows improved results in
terms of CPU, does it also improve overall throughput? If having access to
a physical Linux server would somehow assist, I can arrange to help you
test if you'd like.

Thank you!

Post by Marat Abrarov
Hi namreeb,

some internal mutex which essentially prevents epoll() from executing

across multiple threads simultaneously.
I believe, io_service-per-thread (number of threads ~ number of CPUs)
should solve this. I use this way in asio_samples project (
https://github.com/mabrarov/asio_samples) - the echo_server application
1. single instance of boost::asio::io_service for all work (IO) threads -
default configuration on Windows as Windows IOCP schedules threads better
this way.
2. io_service-per-thread (number of threads == number of CPUs) - default
configuration on non-Windows OS.
Unfortunately, I have no hardware machine (only virtual machine) to test
performance on Linux (the second option). On Linux VM the second option
shows better and more stable CPU loading then the first one.
Regards,
Marat Abrarov.
------------------------------------------------------------------------------
_______________________________________________
asio-users mailing list
https://lists.sourceforge.net/lists/listinfo/asio-users
_______________________________________________
Using Asio? List your project at
http://think-async.com/Asio/WhoIsUsingAsio

[Marat Abrarov]

namreeb,

When you say that a Linux VM with this solution shows improved results in terms of CPU, does it also improve overall throughput?

Throughput / performance cannot be measured on VM. Even the same application under the same conditions can show very different results (up to twice different).

If having access to a physical Linux server would somehow assist, I can arrange to help you test if you'd like.

You can take asio_samples (https://github.com/mabrarov/asio_samples) - asio_performance_test_client and echo_server projects - and measure yourself. Build instructions (cmake) can be found in build/cmake dir.

Regards,
Marat Abrarov.

[Svante Karlsson]

Just as a side note to Marats comments.

1) It's often that other things in the application are the bottleneck than
the raw speed of asio. Then the overhead (or not) of asio under linux is
not that important
2) It's easy to fall into the trap that the actual locking (not in asio -
but in your application due to mutexes) becomes the real problem

With those things said I often try a (mostly) single threaded approach
(like node-js) and then run many such servers on the same node. You will
need a load balancer in front but the solution scales nicely on physical
hardware and on mesos. The big win is no locking in the application that
sometimes makes a big difference and I believe it to be nice to the
processor cache since there is no need to synchronize shared state between
cores.

Post by Marat Abrarov
namreeb,

Post by namreeb
When you say that a Linux VM with this solution shows improved results

in terms of CPU, does it also improve overall throughput?
Throughput / performance cannot be measured on VM. Even the same
application under the same conditions can show very different results (up
to twice different).

Post by namreeb
If having access to a physical Linux server would somehow assist, I can

arrange to help you test if you'd like.
You can take asio_samples (https://github.com/mabrarov/asio_samples) -
asio_performance_test_client and echo_server projects - and measure
yourself. Build instructions (cmake) can be found in build/cmake dir.
Regards,
Marat Abrarov.
------------------------------------------------------------------------------
_______________________________________________
asio-users mailing list
https://lists.sourceforge.net/lists/listinfo/asio-users
_______________________________________________
Using Asio? List your project at
http://think-async.com/Asio/WhoIsUsingAsio