multiprocessing - Python - Accessing sys.stdin from within a class does not work -

multiprocessing - Python - Accessing sys.stdin from within a class does not work -

so trying read piped input sys.stdin within class. problem not getting input stdin while within class can info outside class in main().

is there way access sys.stdin within multiprocessing class ?

here code:

class bufferreader(process): def __init__(self, queue, lock): super(bufferreader, self).__init__() self.queue = queue # number of lines store in buffer before sending processes self.buffer_size = 200000 self.lines_buffer = [] self.lock = lock def run(self): count = 0 try: # each line in stdin line in sys.stdin: # strip line whitespace stripped = line.strip() # if end of line, break if not stripped: break # add together line buffer self.lines_buffer.append(stripped) # if buffer full, process data, , empty buffer if count == self.buffer_size: self.lock.acquire() self.queue.put(self.lines_buffer) self.lock.release() del self.lines_buffer[:] count = 0 # increment line counter count += 1 except keyboardinterrupt: sys.stdout.flush() pass def parse(index, data_queue, lock): while not data_queue.empty(): lock.acquire() if data_queue.empty(): lock.release() sys.exit(0) result = data_queue.get() lock.release() codecs.open("proc-%d" % index, 'w', 'utf-8') fp: line in result: fp.write(line) fp.close() sys.exit(0) def main(): data_queue = queue() lock = lock() br = bufferreader(data_queue, lock) br.start() # spawn processes procs = [process(target=parse, args=(i, data_queue, lock)) in range(5)] p in procs: p.start() br.join() p in procs: p.join() if __name__ == '__main__': main()

with multiprocessing spawn workers in separate processes, own process ids , such including own input , output devices. means sys.stdin/stdout instance obtain within spawned process not same of master process, although can still read , write them.

there @ to the lowest degree 2 options of how go this:

pass sys.stdin/stdout.fileno() file descriptor of master process downwards spawned processes. should able open within spawned processes using os.fdopen(fileno).

use threading instead, threads of same process share input , output devices.

also, pointed out in comments below, reading single input stream concurrently multiple processes may tricky, if don't know doing. wise designate 1 process read input , dispatch info other workers. or introduce kind of round-robin scheme ensure 1 of processes @ time grab input data. process pooling using multiprocessing.pool may come in handy this.

and i'd recommend using fileinput module create rading standard input easier.

python multiprocessing stdin