Shell scripting for daily tasks

Shell scripting empowers IT specialists and software engineers to execute a range of automated tasks via the command line or terminal, such as code compilation or batch processing of computers. In my professional career, I frequently utilize shell scripting for my Django environment or for the creation and activation of virtual environments in Python. However, the appeal of shell scripting extends beyond server operators and programmers. In fact, numerous everyday tasks can be automated through shell scripting, streamlining life's complexities.

Consider the routine task of data backups. As a data scientist or software engineer, maintaining regular data backups is imperative to mitigate the risk of losing extensive lines of reusable code and safeguarding large amounts of private data, including pictures, ebooks, documents, movies and more.

While standard backup software like Déjà Dup on Ubuntu or Time Machine on Mac exists, they often prove to be inflexible and limited in defining what data to store, in manipulating the data or in defining a bunch of parameters that allow for more customization. Shell scripting provides a solution to customize your backup process by creating automated data backup scripts that are reusable and easily adaptable for different backup scenarios. In this post, I outline a straightforward backup shell script for Ubuntu, compatible with UNIX-based systems.

To begin with, I connect my external drive Elements to my Ubuntu laptop and create a new text file to be saved as Backup_012022.sh. Afterward, I right-click on the file, access Properties, navigate to the Permissions tab, and check the box to Allow executing file as a program.

Opening the file in a text editor, I input the shell script. In this script, I first create a new directory on my external drive named Backup_012022 within the Backup folder. I then navigate to the folder containing my private data. To reduce file size and save storage capacity, I compress the folder by zipping it and save the compressed file to my external drive.

mkdir /media/christopher/Elements/Backup/Backup_012022
cd /home/christopher/
zip -r Python_Environment_Backup.zip ./environments
mv Python_Environment_Backup.zip /media/christopher/Elements/Backup/Backup_012022
zip -r Documents_Backup.zip ./Documents
mv Documents_Backup.zip /media/christopher/Elements/Backup/Backup_012022
zip -r Pictures_Backup.zip ./Pictures
mv Pictures_Backup.zip /media/christopher/Elements/Backup/Backup_012022
zip -r Developing_Backup.zip ./Developing
mv Developing_Backup.zip /media/christopher/Elements/Backup/Backup_012022

In the next line of code we zip the next folder, but first exclude another folder x86_64-pc-linux-gnu-library which is located in this folder.

zip -r R_Backup.zip ./R -x "*x86_64-pc-linux-gnu-library*"
mv R_Backup.zip /media/christopher/Elements/Backup/Backup_012022

These steps are repeated for all other folders before copying the shell script from my laptop to the external drive:

cp /home/christopher/Backup_012022.sh /media/christopher/Elements/Backup/Backup_012022

Finally, we save the file, open the terminal and type sh Backup_012022.sh to execute the program and to start the backup process.

This might be the simplest version of a data backup shell script, but it is a good starting point to build more complex scripts and applications. For instance, we could simply change the compression parameters of the zip files or change the compression method from zip to tar and use gzip to store our data. We could also delete older backup folders or use logical operators to build more complex directories. We might also want to encrypt our data, which could be implemented by using encfs.

Shell scripts do not only allow us to automatize simple or more complex processes and reduce time, they also have the advantage that the programs can be customized and designed according to our individual needs and preferences.