It’s quite uncomplicated to read the contents of a Linux textual content file line by line in a shell script—as extended as you offer with some subtle gotchas. Here’s how to do it the safe way.
Table of Contents
Information, Textual content, and Idioms
Every programming language has a established of idioms. These are the conventional, no-frills strategies to execute a established of prevalent tasks. They’re the elementary or default way to use a single of the functions of the language the programmer is functioning with. They grow to be section of a programmer’s toolkit of mental blueprints.
Actions like studying data from files, doing the job with loops, and swapping the values of two variables are fantastic illustrations. The programmer will know at minimum one way to achieve their ends in a generic or vanilla style. Most likely that will suffice for the necessity at hand. Or maybe they’ll embellish the code to make it extra successful or relevant to the distinct option they are creating. But getting the constructing-block idiom at their fingertips is a great beginning issue.
Figuring out and being familiar with idioms in one language tends to make it less difficult to pick up a new programming language, also. Knowing how items are produced in one language and wanting for the equivalent—or the closest thing—in an additional language is a superior way to enjoy the similarities and discrepancies involving programming languages you previously know and the just one you’re understanding.
Reading Lines From a File: The A single-Liner
In Bash, you can use a though
loop on the command line to read every line of textual content from a file and do one thing with it. Our text file is named “data.txt.” It holds a record of the months of the yr.
January February March . . October November December
Our easy a person-liner is:
when go through line do echo $line performedThe
while
loop reads a line from the file, and the execution flow of the little program passes to the body of the loop. Theecho
command writes the line of text in the terminal window. The read attempt fails when there are no more lines to be read, and the loop is done.One neat trick is the ability to redirect a file into a loop. In other programming languages, you’d need to open the file, read from it, and close it again when you’d finished. With Bash, you can simply use file redirection and let the shell handle all of that low-level stuff for you.
Of course, this one-liner isn’t terribly useful. Linux already provides the
cat
command, which does exactly that for us. We’ve created a long-winded way to replace a three-letter command. But it does visibly demonstrate the principles of reading from a file.That works well enough, up to a point. Suppose we have another text file that contains the names of the months. In this file, the escape sequence for a newline character has been appended to each line. We’ll call it “data2.txt.”
Januaryn Februaryn Marchn . . Octobern Novembern DecembernLet’s use our one-liner on our new file.
while read line do echo $line doneThe backslash escape character ”
” has been discarded. The result is that an “n” has been appended to each line. Bash is interpreting the backslash as the start of an escape sequence. Often, we don’t want Bash to interpret what it is reading. It can be more convenient to read a line in its entirety—backslash escape sequences and all—and choose what to parse out or replace yourself, within your own code.
If we want to do any meaningful processing or parsing on the lines of text, we’ll need to use a script.
Reading Lines From a File With a Script
Here’s our script. It’s called “script1.sh.”
#!/bin/bash Counter=0 while IFS='' read -r LinefromFile || [[ -n "$LinefromFile" ]] do ((Counter++)) echo "Accessing line $Counter: $LinefromFile" done "$1"
We set a variable called
Counter
to zero, then we define ourwhile
loop.The first statement on the while line is
IFS=''
.IFS
stands for internal field separator. It holds values that Bash uses to identify word boundaries. By default, the read command strips off leading and trailing whitespace. If we want to read the lines from the file exactly as they are, we need to setIFS
to be an empty string.We could set this once outside of the loop, just like we’re setting the value of
Counter
. But with more complex scripts—especially those with many user-defined functions in them—it is possible thatIFS
could be set to different values elsewhere in the script. Ensuring thatIFS
is set to an empty string each time thewhile
loop iterates guarantees that we know what its behavior will be.We’re going to read a line of text into a variable called
LinefromFile
. We’re using the-r
(read backslash as a normal character) option to ignore backslashes. They’ll be treated just like any other character and won’t receive any special treatment.There are two conditions that will satisfy the
while
loop and allow the text to be processed by the body of the loop:
read -r LinefromFile
: When a line of text is successfully read from the file, theread
command sends a success signal to thewhile
, and thewhile
loop passes the execution flow to the body of the loop. Note that theread
command needs to see a newline character at the end of the line of text in order to consider it a successful read. If the file is not a POSIX compliant text file, the last line may not include a newline character. If theread
command sees the end of file marker (EOF) before the line is terminated by a newline, it will not treat it as a successful read. If that happens, the last line of text will not be passed to the body of the loop and will not be processed.[ -n "$LinefromFile" ]
: We need to do some extra work to handle non-POSIX compatible files. This comparison checks the text that is read from the file. If it isn’t terminated with a newline character, this comparison will still return success to thewhile
loop. This ensures that any trailing line fragments are processed by the body of the loop.
These two clauses are separated by the OR logical operator ” ||
” so that if either clause returns success, the retrieved text is processed by the body of the loop, whether there is a newline character or not.
In the body of our loop, we’re incrementing the Counter
variable by one and using echo
to send some output to the terminal window. The line number and the text of each line are displayed.
We can still use our redirection trick to redirect a file into a loop. In this case, we’re redirecting $1, a variable that holds the name of the first command line parameter that passed to the script. Using this trick, we can easily pass in the name of the data file that we want the script to work on.
Copy and paste the script into an editor and save it with the filename “script1.sh.” Use the chmod
command to make it executable.
chmod +x script1.sh
Let’s see what our script makes of the data2.txt text file and the backslashes contained within it.
./script1.sh data2.txt
Every character in the line is displayed verbatim. The backslashes are not interpreted as escape characters. They’re printed as regular characters.
Passing the Line to a Function
We’re still just echoing the text to the screen. In a real-world programming scenario, we’d likely be about to do something more interesting with the line of text. In most cases, it is a good programming practice to handle the further processing of the line in another function.
Here’s how we could do it. This is “script2.sh.”
#!/bin/bash
Counter=0
function process_line()
echo "Processing line $Counter: $1"
while IFS='' read -r LinefromFile || [[ -n "$LinefromFile" ]] do
((Counter++))
process_line "$LinefromFile"
done "$1"
We define our Counter
variable as before, and then we define a function called process_line()
. The definition of a function must appear before the function is first called in the script.
Our function is going to be passed the newly read line of text in each iteration of the while
loop. We can access that value within the function by using the $1
variable. If there were two variables passed to the function, we could access those values using $1
and $2
, and so on for more variables.
The while
loop is mainly the same. There is only one change inside the body of the loop. The echo
line has been replaced by a call to the process_line()
function. Note that you don’t need to use the “()” brackets in the name of the function when you are calling it.
The name of the variable holding the line of text, LinefromFile
, is wrapped in quotation marks when it is passed to the function. This caters for lines that have spaces in them. Without the quotation marks, the first word is treated as $1
by the function, the second word is considered to be $2
, and so on. Using quotation marks ensures that the entire line of text is handled, altogether, as $1
. Note that this is not the same $1
that holds the same data file passed to the script.
Because Counter
has been declared in the main body of the script and not inside a function, it can be referenced inside the process_line()
function.
Copy or type the script above into an editor and save it with the filename “script2.sh.” Make it executable with chmod
:
chmod +x script2.sh
Now we can run it and pass in a new data file, “data3.txt.” This has a list of the months in it, and one line with many words on it.
January February March . . October November nMore text "at the end of the line" December
Our command is:
./script2.sh data3.txt
The lines are read from the file and passed one by one to the process_line()
function. All the lines are displayed correctly, including the odd one with the backspace, quotation marks, and multiple words in it.
Building Blocks Are Useful
There’s a train of thought that says that an idiom must contain something unique to that language. That’s not a belief that I subscribe to. What’s important is that it makes good use of the language, is easy to remember, and provides a reliable and robust way to implement some functionality in your code.