top button
Flag Notify
    Connect to us
      Site Registration

Site Registration

Running a command line program and reading the result as it runs in Python

+1 vote
469 views

I'm using Python 2.7 under Windows and am trying to run a command line program and process the programs output as it is running. A number of web searches have indicated that the following code would work.

import subprocess

p = subprocess.Popen("D:PythonPython27Scriptspip.exe list -o",
 stdout=subprocess.PIPE,
 stderr=subprocess.STDOUT,
 bufsize=1,
 universal_newlines=True,
 shell=False)
for line in p.stdout:
 print line

When I use this code I can see that the Popen works, any code between the Popen and the for will run straight away, but as soon as it gets to the for and tries to read p.stdout the code blocks until the command
line program completes, then all of the lines are returned. Does anyone know how to get the results of the program without it blocking?

posted Aug 22, 2013 by Satish Mishra

Share this question
Facebook Share Button Twitter Share Button LinkedIn Share Button

2 Answers

+1 vote

Is the program actually producing output progressively? I just tried your exact code with "dir /ad /s /b" and it worked fine, producing output while the dir was still spinning (obviously setting shell=True to make that work, but I don't think that'll make a difference). It may be that pip buffers its output. Is there a parameter to pip to make it pipe-compatible?

answer Aug 22, 2013 by Deepak Dasgupta
If I run pip in the command window I can see it's output appearing line  by line rather than on one block.

I tried the code with the dir command but it's too fast for me to be sure if it's working or not.

I tried again using the command "ping google.com" instead since I know that output's slowly and it something that everyone should have. In the command window I can see that the output appears over time, but from python I get nothing for a while and then suddenly get all the output in one rapid go.

Can you think of anything else I can look at?
A lot of programs, when their output is not going to the console, will buffer output. It's more efficient for many purposes. With Unix utilities, there's often a parameter like --pipe or --unbuffered that says "please produce output line by line", but Windows ping doesn't have that - and so I'm seeing the same thing you are.

You should be able to see the time delay in dir by looking for some particular directory name, and searching from the root directory. Unless you're on a BLAZINGLY fast drive, that'll take Windows a good while!
+1 vote

When file object is used in a for loop it works like an iterator and then it uses a hidden read-ahead buffer.
It might cause this kind of blocking. You can read more details here (description of method next):
http://docs.python.org/lib/bltin-file-objects.html

So basically non-blocking loop might look like this:

while True:
 line = p.stdout.readline()
 if not line: break
 print line
answer Aug 22, 2013 by Anderson
Similar Questions
+1 vote

Something that just sends a screen directly to the printer without having to involve Notepad or the like? I am unable to find a search function on the updated Yahoo group.

+2 votes

Any sample example would be helpful, I want to use all files from a directory and want to apply certain operation on it.

+2 votes

I want to install Python on a PC with the 64 bit version of Windows 7. I want Python to be able to use as much as possible of the RAM.

When I install the 64 bit version of Python I find that sys.maxint == 2**31 - 1 Whereas the Pythpon installed on my 64 bit linux system returns sys.maxint == 2**63 - 1.

It looks to me as though 32 and 64 bit versions of Python on 64 bit Windows are both really 32 bit Python, differing only in how they interact with Windows. So I wouldnt expect 64 bit Python running on 64 bit Windows to allow the large data struictures I could have with 64 bit Python running on 64 bit linux.

Is that true?I have spent a couple of hours searching for a definitive description of the difference between the 32 and 64 bit versions of Python for Windows and haven't found anything.

0 votes

I am writing a command line tool in python to generate one time passwords/tokens. The command line tool will have certain sub-commands like --generate-token and --list-all-tokens for example. I want to restrict access to certain sub-commands. In this case, when user tries to generate a new token, I want him/her to authenticate against AD server first.

I have looked at python-ldap and I am even able to bind to the AD server. In my application I have a function

 def authenticate_user(username, password): pass

which gets username and plain-text password. How do I use the LDAPObject instance to validate these credentials?

...