[asio-users] Async HTTP and requests requiring long processingtime

[Gruenke, Matt]

BTW, that's also assuming there are enough threads to handle all
connected client & incoming requests. Obviously, this design is not
going to scale well, but that's not my point.

I'm trying to ask a narrow question about whether this is simply generic
advice about good server design, or whether it comes from knowledge of
anything in the implementation of ASIO, whereby long-running completion
handlers would actually block anything else.

I thought the whole point of io_services supporting multiple threads was
to allow completion handlers to execute in parallel, in which case I'd
expect one not to block anything else.


Matt


-----Original Message-----
From: Gruenke, Matt [mailto:***@Tycoint.com]
Sent: June 14, 2013 15:05
To: asio-***@lists.sourceforge.net
Subject: Re: [asio-users] Async HTTP and requests requiring long
processingtime

Can anyone explain why? Are there any locks held while executing
completion handlers? Or does this advice originate purely from a
standpoint of not wanting to let the connection go idle?


Matt

-----Original Message-----
From: Green, Cliff [mailto:***@boeing.com]
Sent: June 14, 2013 13:47
To: asio-***@lists.sourceforge.net
Subject: Re: [asio-users] Async HTTP and requests requiring long
processing time

... Recently we added a new feature that requires a long processing

time like 10 minutes.

As Guilherme has already recommended, queuing the work to another thread
is the way to go. This is a general recommendation for Asio design - if
a handler is going to take a "long" time to complete, pass the work off
to something else. I generally do all network event processing in one
thread - i.e. all Asio related work is in a single thread (which
simplifies a lot of issues), and if functionality is needed that is
compute or IO intensive, that work is passed to another thread (which
knows nothing about network / Asio). Sometimes the design can get a bit
complicated when dealing with "transactional" type network requests, but
there are straightforward solutions.

Cliff


------------------------------------------------------------------------
------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
asio-users mailing list
asio-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/asio-users
_______________________________________________
Using Asio? List your project at
http://think-async.com/Asio/WhoIsUsingAsio

[Marat Abrarov]

Post by Gruenke, Matt
I thought the whole point of io_services supporting multiple threads was
to allow completion handlers to execute in parallel, in which case I'd
expect one not to block anything else.

At least at Windows it is so ("one handler not to block anything else" but
occupies thread running on asio::io_service). At *nix it has to be so too.
I think the Bruno's problem is related to something else:
1) maybe his thread pool consists of one thread or,
2) maybe the "long running task" was the overall bottleneck (locks, etc.)
for all of the handlers executed by server thread pool.

In general, SEDA or at least multiple (fixed size) thread pools (clients IO,
work, file IO, IO with external systems, etc.) is the most common solution
for scalable server architecture.

Regards,
Marat Abrarov.

[Bruno Coudoin]

Post by Marat Abrarov
1) maybe his thread pool consists of one thread or,

I tried to increase our thread pool with few significant improvements.
For instance I could repreduce the issue with more threads in the pool
than my number of clients.

Post by Marat Abrarov
2) maybe the "long running task" was the overall bottleneck (locks, etc.)
for all of the handlers executed by server thread pool.

I just changed my implementation and perform this task in background,
out of my asio handle request. In both cases this processing takes 100%
of a single core but my PC has much more. If it was a CPU issue my fix
would not have changed the strange behavior.

It is not a lock issue either, in fact the client request don't even get
to my handle request so there is no way for me to lock it.

Bruno.

[Marat Abrarov]

Post by Bruno Coudoin
I tried to increase our thread pool with few significant improvements.
For instance I could repreduce the issue with more threads in the pool
than my number of clients.

A working example would be interesting to investigate. I doubt this is an
issue of Asio...

Regards,
Marat Abrarov.

[Roger Austin (Australia)]

I have seen quite a lot of good design advice on this thread but no answer to Bruno's issue, which is that while his long-running handler is executing, sockets stack up in the CLOSE_WAIT state and ultimately the server stops accepting new connections. His incoming connections continue to work for a while and their handlers run until their sockets reach the CLOSE_WAIT state so it is not a pure blocking issue.

A socket in the CLOSE_WAIT state is waiting for the application to call close on the socket. Unless the application code does this explicitly, the asio socket wrapper will do this in its destructor. My speculation is that the asio socket wrappers are not being destroyed, which suggests that some object is holding a shared_ptr to the connection object that contains the socket wrapper. I suggest that Bruno look at the objects that are bound to his long-running handler to see if they are directly or indirectly holding shared_ptrs to other connection objects. Because new connections (after the long-running handler has been bound) are also affected, perhaps his app is keeping some sort of list of active connections to enable timely server shutdown (the shutdown in HTTP Server 3 allows currently-executing handlers to run to completion, which is not good if you have a long-running handler).

-----Original Message-----
From: Bruno Coudoin [mailto:***@rcsmobility.com]
Sent: Saturday, 15 June 2013 07:16
To: asio-***@lists.sourceforge.net
Subject: Re: [asio-users] Async HTTP and requests requiring long processingtime

Post by Marat Abrarov
1) maybe his thread pool consists of one thread or,

I tried to increase our thread pool with few significant improvements.
For instance I could repreduce the issue with more threads in the pool than my number of clients.

Post by Marat Abrarov
2) maybe the "long running task" was the overall bottleneck (locks,
etc.) for all of the handlers executed by server thread pool.

I just changed my implementation and perform this task in background, out of my asio handle request. In both cases this processing takes 100% of a single core but my PC has much more. If it was a CPU issue my fix would not have changed the strange behavior.

It is not a lock issue either, in fact the client request don't even get to my handle request so there is no way for me to lock it.

Bruno.

[Green, Cliff]

BTW, that's also assuming there are enough threads to handle all connected client & incoming requests.

My (simplistic?) design approach of isolating all Asio and network handling to a single thread is not a recommendation for all designs. Most of my code is used in environments where the network IO is steady, sometimes fairly busy, but not maxed out. I've found that a single thread to handle many connections and associated processing provides better performance than many threads and / or thread pools. This is due to significantly less mutex locking and context switching. So yes, I have enough threads (one) to handle all connected client and incoming requests.

Also specific to my application domain is that most of the processing is not performed in the Asio handler - almost everything is passed on (via a wait queue) to other areas (some with thread pools) to do the "real work". My completion handlers usually do the unmarshalling (sometimes) of a message, then put it in a format that is sent on for further processing. There is very little "receive data, send a reply to the sending client" functionality in my domain, so there is almost nothing equivalent to what would be performed by an HTTP server.

Obviously, this design is not going to scale well, but that's not my point.

My design for my domain scales very well, better than a thread pool. At some point it could reach some kind of connection and processing saturation, but with that much traffic I'm not sure a thread pool would help, rather partitioning into multiple executables (which is already supported in my domain).

I'm trying to ask a narrow question about whether this is simply generic advice about good server design

Mine is not generic advice, but I always mention it because there seems to be a bias towards always having multiple threads (maybe it's because some developers come from a "thread per connection" design history). For many domains a single thread (even with thousands of connections) might offer better performance and definitely simpler coding. And Asio is specifically designed to support this approach.

or whether it comes from knowledge of anything in the implementation of ASIO, whereby long-running completion handlers would actually block anything else.

For a given thread, a long-running completion handler would always block other completion handlers from executing on that thread, unless I'm completely mistaken. (I haven't looked much into the Asio implementation.)

I thought the whole point of io_services supporting multiple threads was to allow completion handlers to execute in parallel, in which case I'd expect one not to block anything else.

I assume so, but this is deeper in Asio implementation territory than I know.

Cliff