- Python | Execute and parse Linux commands
- Subprocess –
- Starting a process –
- Listing the directories –
- How to Execute Linux Commands in Python
- Table of contents
- Building an application to ping servers
- Best Way To Execute Linux Commands using Python
- The OS module in Python
- Accept Linux Command As User Input
- Complete Code for Executing Linux Commands Using Python
- Wrapping Up!
- Executing Shell Commands with Python
- Execute Shell command in Python with os module
- Execute shell command in Python with subprocess module
Python | Execute and parse Linux commands
Linux is one of the most popular operating systems and is a common choice for developers. It is popular because it is open source, it’s free and customizable, it is very robust and adaptable.
An operating system mainly consists of two parts: The kernel and the Shell. The kernel basically handles communication between the software and the hardware. The shell takes inputs or commands from the user and produces an output. Most Linux distributions nowadays use the BASH shell (Bourne again shell). Shell commands and scripts are very powerful and are used commonly by developers.
In this article, we shall look at executing and parsing Linux commands using python.
Subprocess is a module in Python that allows us to start new applications or processes in Python. This module intends to replace several older modules in python. We can use this module to run other programs or execute Linux commands.
Starting a process –
A new process can be spawned by using the Popen function defined in the subprocess module. It is a constructor for the Popen class that takes arguments to set up the new process. The underlying process creation and management in this module is handled by the Popen class.
The first parameter is a list that contains the commands and their options if any.
ex: [‘ls’, ‘-l’]
the above example is equivalent to typing ‘ls -l’ in the terminal
The second parameter is the stdout value. it specifies the standard output.
ex: stdout = subprocess.PIPE
This indicates that a new pipe or redirection should be created. The default value is
“None”, which means that no redirection will occur.
We can retrieve the output of a command by using the communicate function. It reads data from stdout and stderr until it reaches the end-of-file and waits for the process to terminate. It returns a tuple that contains the output data and the error if any.
The output of the executed command is stored in data. Using these functions, we can execute Linux commands and fetch their output.
Listing the directories –
We can use the ‘ls’ command with options such as ‘-l’, ‘-al’, etc to list all the files in the current directory. We can then parse this output and print it in a presentable format. The get_permissions() function parses the output of the list command and retrieves only the names of the files and their corresponding permissions.
How to Execute Linux Commands in Python
January 13, 2021
Linux is one of the most popular operating systems used by software developers and system administrators. It is open-source, free, customizable, very robust, and adaptable. Making it an ideal choice for servers, virtual machines (VMs), and many other use cases.
Therefore, it is essential for anyone working in the tech industry to know how to work with Linux because it is used almost everywhere. In this tutorial, we are going to look at how we can automate and run Linux commands in Python.
Table of contents
- Basic understanding of Linux and shell scripting.
- Basic programming skills in Python.
Python has a rich set of libraries that allow us to execute shell commands.
A naive approach would be to use the os library:
The os.system() function allows users to execute commands in Python. The program above lists all the files inside a directory. However, we can’t read and parse the output of the command.
In some commands, it is imperative to read the output and analyze it. The subprocess library provides a better, safer, and faster approach for this and allows us to view and parse the output of the commands.
|os.system function has been deprecated. In other words, this function has been replaced.||The subprocess module serves as a replacement to this and Python officially recommends using subprocess for shell commands.|
|os.system directly executes shell commands and is susceptible to vulnerabilities.||The subprocess module overcomes these vulnerabilities and is more secure.|
|The os.system function simply runs the shell command and only returns the status code of that command.||The subprocess module returns an object that can be used to get more information on the output of the command and kill or terminate the command if necessary. This cannot be done in the os module.|
Although you can execute commands using the OS module, the subprocess library provides a better and newer approach and is officially recommended. Therefore, we are going to use subprocess in this tutorial. This documentation explores the motivation behind creating this module.
Building an application to ping servers
Let’s use the subprocess library to write a script that pings multiple servers to see whether they are reachable or not. This would be a good use case when you have multiple hosts, servers, or VMs(AWS ec2 instances) and want to check if they are up and running without any problems.
A simple solution is to just ping these servers and see if they respond to the request. However, when you have a considerable amount of machines, it will be extremely tedious and time-consuming to manually ping them. A better approach is to use Python to automate this process.
According to the official documentation, the subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes.
This module intends to replace several older modules and functions. The subprocess library has a class called Popen() that allows us to execute shell commands and get the output of the command.
Create a Python file and add the following code. We also need to create a file called “servers.txt”, where we can add a list of all the servers we need to ping. The Python script will read from this file and ping each server listed in it.
I have added 4 servers, out of which two exist and the other two do not. Only the servers that exist can be “pinged”.
As you can see in the output, we get the message “name or service not known” for the two servers that did not exist.
In the program above, the ping() function takes a list of servers and returns the output of each running ping command on each server. If a server is unreachable, it displays an output saying “ping: somethingthatdoesntexist: Name or service not known”.
The Popen() is a constructor method of the Popen class and takes in the following arguments:
A list of commands and any additional options these commands might require. For example, the ls command can be used with ‘-l’ option. To execute the ls -l command, the argument list would look like this: [‘ls’, ‘-l’] . The commands are specified as strings. In the example above, we use the ping command with the option -c 1 so that it only sends one packet of data, and the server replies with a single packet . Without this limit, the command would run forever until an external process stops it.
The stdout argument is optional and can be used to set where you want the subprocess to display the output. By default, the output is sent to the terminal. However, if you don’t want to dump a large output onto the terminal, you can use subprocess.PIPE to send the output of one command to the next. This corresponds to the | option in Linux.
The stderr argument is also optional and is used to set where you want the errors to be displayed. By default, it sends the errors to the terminal. Since we need to get a list of servers that cannot be reached, we don’t need to change this. The servers that cannot be reached (error) will be displayed to us on the terminal.
The output of the command is stored in a variable called temp . The communicate() function allows us to read the output and the str function can be used to convert it to a string. Once we get the output, we can parse it to extract only the essential details or just display it as it is. In this example, I am storing the output in a list for future use.
In conclusion, automation is one of the hottest topics in the industry, and almost every company is investing huge amounts of money to automate various manual tasks. In this tutorial, we explored the process of automatically running and analyzing Linux commands on multiple hosts using Python.
An old way of doing this is by using shell scripts. However, using Python gives developers more power and control over the execution and output of the commands. Now that you have understood the basics of executing Linux commands, you can go ahead and experiment with different commands and build more complex and robust applications.
Best Way To Execute Linux Commands using Python
Python is a trendy & modern programming language. Especially popular among beginners as it is easy to learn. Python is so powerful that we can use it to interact with the operating system. In Linux, python act as an alternative for bash command language for scripting. It comes preinstalled in most of the distributions as it is a dependency on many tools and software. And if not it is effortless to install.
In this tutorial, we will see how to run Linux commands using Python. The idea behind this is to automate tasks and save time. I am using Ubuntu for this tutorial.
Table of Contents
We will require a python environment. If you are using the latest distribution it is preinstalled. You can check using the following command.
To install python run the following command –
Additionally, if you have some knowledge of terminal & some Linux commands that will be great.
The OS module in Python
OS library provides operating system-based functions and allows us to interact with the operating system. The package comes with python installation, so no need to install it separately. The library is easy to use and contains both read and write operations.
Let’s write our first program and get system information –
Launch your favorite text editor (I am using gedit) –
Copy the below code and save the file –
You can run the file like the below –
Get System Info
The uname() method returns information like name, release, and version of the current operating system, etc. This was just an overview of the OS module, now let’s run the terminal commands.
The os.system() method takes command as a string and executes it in a subshell. The limitations stand the same as the system() method in C language. The method is system dependent.
The code should be executed using the terminal as if you run the code in IDLE you get “0” as output and nothing really happens.
Let’s run our very first command –
list files and folders
The ls command is used to list items in the respective directory.
Accept Linux Command As User Input
Let’s take input from users and then concatenate it with our command. For eg: Creating folders while taking folder names from the user.
Complete Code for Executing Linux Commands Using Python
I have developed a simple script to give you guys an idea of how you can use python to make your job easy.
This will show a menu to the user and the user has to choose any one option. As the script is in a while loop the menu will be presented to the user continuously. You can add more options to the menu.
The Script Of Everything
This was just a small part of the OS module. It has many more features and methods. You can set the path of the scripts you have developed in the shell configuration file. This will allow you to run the scripts from anywhere. Also, create cron jobs to execute automatically from time to time. Hope this article helped you in some way.
Executing Shell Commands with Python
Python is an excellent scripting language. More and more sysadmins are using Python scripts to automate their work.
Since the sysadmin tasks involve Linux commands all the time, running Linux commands from the Python script is a great help.
In this tutorial, I’ll show you a couple of ways you can run shell commands and get its output in your Python program.
Execute Shell command in Python with os module
Let me create a simple python program that executes a shell command with the os module.
Now, if I run this program, here’s what I see in the output.
That’s the content of the directory where prog.py is stored.
If you want to use the output of the shell command, you can store it in a file directly from the shell command:
You can also store the output of the shell command in a variable in this way:
If you run the above program, it will print the content of the variable myCmd and it will be the same as the output of the ls command we saw earlier.
Now let’s see another way of running Linux command in Python.
Execute shell command in Python with subprocess module
A slightly better way of running shell commands in Python is using the subprocess module.
If you want to run a shell command without any options and arguments, you can call subprocess like this:
The call method will execute the shell command. You’ll see the content of the current working directory when you run the program:
If you want to provide the options and the arguments along with the shell command, you’ll have to provide them in a list.
When you run the program, you’ll see the content of the current directory in the list format.
Now that you know how to run shell command with subprocess, the question arises about storing the output of the shell command.
For this, you’ll have to use the Popen function. It outputs to the Popen object which has a communicate() method that can be used to get the standard output and error as a tuple. You can learn more about the subprocess module here.
When you run the program, you’ll see the stdout and stderr (which is none in this case).
I hope this quick tip helped you to execute shell command in Python programs. In a related quick tip, you can learn to write list to file in Python.
If you have questions or suggestions, please feel free to drop a comment below.