Loop over file names without using ls

This is a common misconception coming from old bash versions or old shells altogether. When executing loops over file names, the for loop has direct support for path expansion which means you don’t have to use ls in a subshell to obtain the same result. The form using a subshell This form is very commonly used, even though it uses a subshell and it’s less visually appealing – obviously, this is personal opinion. for FILE in `ls /etc/*rel*`; do … $FILE … ; done Spawning a subshell that executes ls can be expensive and most probably the reason why this form is so common isRead More →

Simple realtime blocking of failed login attempts

I get this question a lot and notice many folks using complex scripts for a problem which is, even in enterprise environments, a simple one. How do you ban failed login attempts in an efficient way? The idea is simple. Some daemon detects failed ssh attempts and feeds those IP addresses to a network access control system, which in turn denies the access for a while. That’s it. Solution features The solution that I’m describing in this post, I thought of it while trying to solve this annoying problem. And even though it was on everyone’s lips, I found fail2ban far too complicated for theRead More →

Pipelines and their quirks

Pipelines are an amazing feature of bash. I use them all the time. And I noticed a lot of situations when the shell code abuses the pipelines. It’s this abuse that’s bothering me so much that I decided to write a post exclusively about how to use pipelines properly. What are pipelines? A pipeline connects the standard output of a command to the standard input of the next. Assuming you want to print out the names of all users with a /bin/bash shell, a way of achieving this is: grep /bin/bash /etc/passwd | cut -d: -f1 The | character signifies where the first command ends,Read More →

Measuring execution time with bash (part 2/2)

In the first part of this topic, we’ve seen what bash (and GNU) offers in terms of commands specially designed to measure the execution time and resource consumption. However, in some situations, that’s not enough, and you have to measure the execution time manually. You can achieve this in two similar ways. Get the epoch time at the start and the end of the execution block and compute the difference. Let’s see it in action. What’s epoch you’re asking me? It’s the number of seconds since 1st of January 1970. About 1.5 billion seconds so far, so we’re good. printf builtin In bash 4.4 orRead More →

Measuring execution time with bash (part 1/2)

Things seem comfortable when you want to measure the execution time of a single command at a shell prompt. You can even use your phone’s chronograph for this. However, what are your options when you want to automate this task and your code is a lot more complex? Read on! Time reserved word The time reserved word is suitable for being used when you have a simple command. Because it’s a reserved word, it can also compute times for a pipeline. Another advantage is that you also get the system and kernel CPU times. That’s useful when your code does some advanced testing of theRead More →

Bash 5.0. Main features and where it's included

We’re well into 2019 now, but right at the beginning bash version 5.0 was released. Significant version, since it’s also a major increment, but also with lots of new features and bug fixes included. I’ll write separate posts about some of these features and how we can use them. My favourite bash 5.0 features For now, let me tell you which major features included in bash 5.0 I find exciting. Environment variables Let’s start with the three new environment variables. BASH_ARGV0 is a new variable that allows you to set the zeroth argument of a shell script. Of course, this is useful for debugging orRead More →

Replace date with printf

I get it, you’re attached to date. It served you well. All these tens of years we have used this simple yet effective binary to give meaning to our logs and backups. Those days are gone. Now, you can do all that in pure bash. Interested? Well, the title did give it away, so… I present you: printf. First, let’s explore how this printf is a replacement. Starting with bash 4.2, the date format feature was added to printf. Since it was released in 2011, I can only assume that most currently supported distros out there have this feature. I haven’t met one that didn’t.Read More →

Practical pathname expansion

One of the most common situations where folks get confused is when they want to iterate through all or some of the files inside a certain directory. Iterate = a for loop, files list = ls. Right? Nope! The correct answer is the pathname expansion. I challenge you, look through the results from a GitHub search of for in ls. It wasn’t hard for me to find out examples for this blog post. And reasons why folks fail to understand this essential bash feature. What is it? So what is pathname expansion? It happens along with most of the other types of expansion, right afterRead More →

Displaying unique values in pure bash

I have done it, you have done it, we’re all guilty of piping the output of some command to sort | uniq. What if I tell you that you can achieve value uniqueness by using pure bash code? The sort|uniq cases In general, when piping the commands through the sort | uniq pair we’re not really interested in the sorting part. Most of the times we’re interested in the uniqueness part. But uniq can only function correctly if all the duplicates are one after another in the output. And this behaviour is achieved with the help of the additional sort call beforehand. Nowadays sort evenRead More →