multithreading - python requests-futures slow - not threading properly? -

multithreading - python requests-futures slow - not threading properly? -

hi have written multithreaded request , response handler using requests-futures library.

however, seems slow, , not asynchronous imagine. output slow , in order, not interlaced expect if threading properly.

my question why code slow, , can speed up? illustration great.

here code:

#!/usr/bin/python import requests import time concurrent.futures import threadpoolexecutor requests_futures.sessions import futuressession session = futuressession(executor=threadpoolexecutor(max_workers=12)) def responsecallback( sess, resp ): response = resp.text if not "things invalid" in response in response: resp.data = "success %s" % resp.headers['content-length'] else: resp.data = "fail %s" % resp.headers['content-length'] proxies = { "http":"http://localhost:8080", "https":"https://localhost:8080" } url = 'https://www.examplehere.com/blah/etc/' headers= { 'host':'www.examplehere.com', 'connection':'close', 'cache-control':'max-age=0', 'accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'origin':'https://www.examplehere.com', 'user-agent':'mozilla/5.0 (x11; linux x86_64) applewebkit/533.32 (khtml, gecko) ubuntu chromium/34.0.1847.123 chrome/34.0.1847.123 safari/337.12', 'content-type':'application/x-www-form-urlencoded', 'referer':'https://www.exampleblah.etc/', 'accept-encoding':'gzip,deflate,sdch', 'accept-language':'en-us,en;q=0.8,de;q=0.6', 'cookie':'blah=123; etc=456;', } n in range( 0, 9999 ): #wibble = n.zfill( 4 ) wibble = "%04d" % n payload = { 'name':'test', 'gennum':wibble, 'button1':'push+now' } #print payload #r = requests.post( url, data=payload, headers=headers, proxies=proxies, verify=false ) future = session.post( url, data=payload, headers=headers, verify=false, background_callback=responsecallback ) response = future.result() print( "%s : %s" % ( wibble, response.data ) )

ideally i'd prepare actual code still using library have utilised, if it's bad reason i'm open suggestions...

edit: using python2 concurrent.futures backport.

edit: slow - approx 1 request second, , not concurrent, 1 after other, request1, response1, request2, response2 - expect them interlaced requests go out , come in on multiple threads?

the next code way submit multiple requests, work on several of them @ time, print out results. results printed ready, not in same order when submitted.

it uses extensive logging, help debug issues. captures payload logging. multithreaded code hard, more logs more better!

source import logging, sys import concurrent.futures cf requests_futures.sessions import futuressession url = 'http://localhost' num = 3 logging.basicconfig( stream=sys.stderr, level=logging.info, format='%(relativecreated)s %(message)s', ) session = futuressession() futures = {} logging.info('start') n in range(num): wibble = "%04d" % n payload = { 'name':'test', 'gennum':wibble, 'button1':'push+now' } future = session.get( url, data=payload ) futures[future] = payload logging.info('requests done, waiting responses') future in cf.as_completed(futures, timeout=5): res = future.result() logging.info( "wibble=%s, %s, %s bytes", futures[future]['gennum'], res, len(res.text), ) logging.info('done!') output 69.3101882935 start 77.9430866241 starting new http connection (1): localhost 78.3731937408 requests done, waiting responses 79.4050693512 starting new http connection (2): localhost 84.498167038 wibble=0000, <response [200]>, 612 bytes 85.0481987 wibble=0001, <response [200]>, 612 bytes 85.1981639862 wibble=0002, <response [200]>, 612 bytes 85.2642059326 done!

python multithreading asynchronous python-requests