Python-Subprocess

Before I start the topic, I would like to start with the real-time scenario…

I have gone throw challenges on managing the people who are called by me when I planned to do alteration at my home.

  1. Sometimes I don’t know whether they are working,
  2. How they are working,
  3. what is the progress on work,
  4. how many hours they are taking to complete the work,
  5. what they want,
  6. why they are not working,
  7. What are they expect to finish the work?

But now it is very easy to manage any kind of people that I called. Mr.Python suggest me a person who’s named is Subprocess. Subprocess is the very decent person handling the people (process) that I wanted to call. I simply instruct subprocess with simple in & out variables.

Now let’s know more about subprocess:

Subprocess is specifically used when there are requirements to call external OS specific commands, external program or any executable program either it is parameterized or user input required to pass on the console for that program.

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.

ls -lrt   -- a simple unix well known commands

Now there are requirements for me to call this command when I do coding in python. There are many ways to call this command but I want to know the best way of calling this command so that I can pass the input that it may be required, fetch the output that it may produce or caught the error that may occur at once spawn it from my coding. Yes, the best way of calling this command is subprocess.

Why?? Below example would give you light

Vinoth.Durairaj@local:~$ ls -lrt
total 16
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins  242 May 21 17:32 local1.py
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 1491 May 21 17:42 local2.sh
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 1809 Jul 10 15:43 local3.sql
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 2663 Jul 16 16:58 local4.sh
Vinoth.Durairaj@wp1rngs02t:~$ echo $? -- previous command exist status
0    --- It gives exist status "0" because previosu command correct.

let me pass the invalid option to get the exit status.

Vinoth.Durairaj@local:~$ ls -z --- passing invalid option to check the exit status
total 16
ls: invalid option -- 'z'
Try `ls --help' for more information.
Vinoth.Durairaj@wp1rngs02t:~$ echo $? -- previous command exist status
2     --- It gives exist status "2" because previosu command failed.

Now let me execute this command from Python as per requirements but this time i am going to use most known way of os.system.

>>> import os
>>> os.system('ls -lrt')
total 16
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins  242 May 21 17:32 local1.py
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 1491 May 21 17:42 local2.py
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 1809 Jul 10 15:43 local3.py
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 2663 Jul 16 16:58 local4.py
0   ---- its executed and notify that the command output also 0.

Let’s pass the same invalid option to the command calling from os.system to know what is the return code it produces to my python code.

>>> os.system('ls -z')    --- Now i am trying to give invalid option to check how truthful the os.command to me.
ls: invalid option -- 'z'
Try `ls --help' for more information.
512     

Oops, what is that 512? this is not the exit status that I am expecting and it is not matching the return code received when I directly executed. |So this is not the best option to use in my python coding.

Ok. So as I said, subprocess is the best way of calling this command. HOW? let’s see!!

>>> import subprocess
>>> subprocess.call(["ls","-lrth"])
total 16
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins  242 May 21 17:32 local1.py
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 1491 May 21 17:42 local2.py
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 1809 Jul 10 15:43 local3.py
-rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 2663 Jul 16 16:58 local4.py
0   ---- its executed and notify that the command output also 0.

no changes in the output when we compare it with os.system.

>>> subprocess.call(["ls","-z"])    --- Now i am trying to give invalid option to check how truthful the os.command to me.
ls: invalid option -- 'z'
Try `ls --help' for more information.
2   

ahh, this is what I expect in my coding. So subprocess is the right option.

I learned that subprocess is the right module to achieve my requirements, so let’s explore more about it, to do so we need to use “dir” to know any module in python as below.

>>> dir(subprocess)
['CalledProcessError', 'MAXFD', 'PIPE', 'Popen', 'STDOUT', 'TimeoutExpired', '_PIPE_BUF', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '_active', '_cleanup', '_demo_posix', '_demo_windows', '_eintr_retry_call', '_has_poll', 'call', 'check_call', 'errno', 'fcntl', 'gc', 'list2cmdline', 'mswindows', 'os', 'pickle', 'select', 'signal', 'sys', 'time', 'traceback', 'types']

we will go through only callable objects.

subprocess.call : As in the example, any process/program/commands can be executed using this method but what if an error occurs, that I wanted to catch as an exception in my code, fine we can use another method called check_call.

subprocess.check_call : I can use this if I directly wanted to jump to an exception. what if I wanted to fetch the output from the command? yes, we can use another method check_output.

>>> subprocess.check_call(["ls","-z"])
ls: invalid option -- 'z'
Try `ls --help' for more information.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/subprocess.py", line 505, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['ls', '-z']' returned non-zero exit status 2

subprocess.check_output : I can use this if I am expecting output or want to pass input to the process/program that I called from my code. If the called process gives output then it will provide the output and if the called process throws an error then it will flow my code to an exception when caught this kind of exception using CalledProcessError.

The next frequently used the method in subprocess is Popen. Popen is very supportive in subprocess for the developer who uses this module and wants to play around with external process signals.

subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)

with this Popen method, I can easily communicate, kill, poll, terminate, send_signal and wait on the process/program that I have called from my python code.

ok. let me try the Popen with same ‘ls’ command.

>>> subprocess.Popen(["ls"])
local1.py local2.py local3.py local4.py 

the output is as expected. but how can I communicate with the process/program?

>>> ret = subprocess.Popen(["ls"])
>>> local1.py local2.py local3.py local4.py
>>> ret.communicate()
(None, None)

communicate is the callable module of subprocess.Popen, which always produce the output in tuple data structure. But in the above output was only (None, None). why?? to make any communication to the external process, we need to first construct the pipe with three phase such as stdin, stdout & stderr.

Ok, let’s see how to make PIPE connection.

>>> ret = subprocess.Popen(["ls","-lrt"], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> ret.communicate()
(b'total 16 -rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins  242 May 21 17:32 local1.py -rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 1491 May 21 17:42 local2.py -rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 1809 Jul 10 15:43 local3.py -rw-r--r-- 1 Vinoth.Durairaj grpAppAdmins 2663 Jul 16 16:58 local4.py', b'')

So here I have made PIPE connection which is an instance of subprocess. Now when I call the communicate, it gives me output in tuple format, where the first set is the output of the command and second set is an error which is blank now. How it’s working?

stdin: to communicate with your process by passing input such parameter, value and user input.

stdout: to communicate with your process until end-of-file / terminate process.

stderr: to communicate with your process until end-of-file / terminate process.

eg: I have the program that expects user to input “1” to say “Hi”, “2” to say “Hello” and “3” to exit. Now I can easily call that program from my python subprocess.Popen to communicate with “1” “2” & “3”.

ret=subprocess.Popen([hifi_say.pl],stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
                        ret.stdin.write('n')
                        ret.stdin.write('n 1')
                        ret.stdin.write('n 7')
out,err = ret.communicate()
print out
Hi

In the above example, I have passed “1” to get “Hi” and “7” to exit that process. This is the simple example of how we can communicate with process/program or commands that we are calling from our python code. Hope this would have given some idea about subprocess. I leave to you rest to explore more about subprocess to play with it in your any kind of development requirements.

1 thought on “Python-Subprocess”

Leave a Reply