Is there a way to find lines shared by all files in a directory?

Question

I need your help for solving a problem in bash. I'm starting to use it and now I need to extract only words (in the second column) in common among all files that I have in a folder. I understood how it works with only two files but not with all of those. This is the beginning of my script:

for file in *
do
    awk '{print $2}' $file | sort -n > ord/$file
done

The above lines worked for extracting and sorting the second column, but now I don't know how can I find only words that are in all files.

Answer 1

Extracting Lines Common For All Files

The following recursive command extracts lines common for all files. An advantage is that we don't have to sort anything.

intersect() { f="$1"; if shift; then grep -Fxf "$f" | intersect "$@"; else cat; fi; }
common() { f="$1"; shift; intersect "$@" < "$f"; }
common *

The trick here is to intersect files recursively. If we understand files as mathematical sets of lines the question boils down to »Given sets a, b, …, n, how to compute a ∩ b ∩ … ∩ n«.

We can compute the intersection a ∩ b with the command grep -Fxf a b which is the same as cat b | grep -Fxf a or cat a | grep -Fxf b (useless use of cat only for better readability). The order of a and b does not matter.

To compute the intersection a ∩ b ∩ c we can compute (a ∩ b) ∩ c. How to compute (a ∩ b) is already known (see above), so we apply the same approach to the result of (a ∩ b): cat a | grep -Fxf b | grep -Fxf c. Alternatively, you can replace the entire grep command by common a b from moreutils.

How to proceed from there should be clear.

Use 2nd Column Instead Of Whole Lines

To use only the 2nd column instead of whole lines we can either work on modified copies

for f in *; do
    awk '{ print $2 }' "$f" > "$f-col2"
done
common *-col2

… or adapt the function

mask() { awk '{ print $2 }' "$@"; }
intersect() { f="$1"; if shift; then grep -Fxf <(mask "$f") | intersect "$@"; else cat; fi; }
common() { f="$1"; shift; mask "$f" | intersect "$@"; }
common *

Answer 2

Here's a simple Awk script to print all values of $2 which are present in all the files.

awk '# Count number of files; no lines were seen in this file yet
    FNR==1 { file++; delete b }
    # If not already seen in this file, add one to count
    # and mark as seen in this file as a side effect
    !b[$2]++ { a[$2]++ }
    # In the end, print all values which occurred in all files
    END { for (k in a) if (a[k]==file) print k }' *

This will examine all files in the current directory. You can replace the wildcard at the end with whatever will match the set of files you want to examine.

With comments removed, this can be a one-liner, though let's not cheat too much. Here's a two-liner:

awk 'FNR==1 { file++; delete b }  !b[$2]++ { a[$2]++ }
    END { for (k in a) if (a[k]==file) print k }' *

Answer 3

Try something like this:

$ FILES=`ls -1 *`
$ COUNT=`grep -c ^ <<<"$FILES"`
$ for FILE in $FILES; do awk '{ print $2}' $FILE | sort -u; done | \
     sort | uniq -c | grep " $COUNT "

Breaking this apart, we first get the list of files into FILES, and then count how many into COUNT - this is used at the end of the process.

Then we get the words in the second column in each file and use "sort -u" to return just one of each.

We do this in a loop for all files, and then count the number of times each word appears. This uses "uniq -c" which displays the words prefixed by the count of words. So if the word "pepper" appears in 7 files, the loop outputs "pepper" once for each of those 7 files, and "uniq -c" outputs " 7 pepper" (always has whitespace at the start). If the total number of files was 7, then we now know every file had at least one instance of the word "pepper" in the second column.

We know that the number of files is in COUNT. So we just search for the "uniq -c" output that has " 7 " (with spaces either side).

Answer 4

Here is another awk one:

awk '(NR==FNR){a[$2]++; next}
     (FNR==1) { for(i in a) if (a[i]==0) delete a[i]; else a[i]=0; }
     ($2 in a) {a[$2]++}
     END { for(i in a) if (a[i]!=0) print i }' f1 f2 f3 f4 ...

This works in the following way. We keep track of an array a which holds all common entries. If a value is seen in a file, we increment the value of the array. Each time a new file is read, we check which values are still zero and delete them from the array:

• (NR==FNR){a[$2]++; next}: the first file is read. Initialise the array a with all its values.
• (FNR==1) { for(i in a) if (a[i]==0) delete a[i]; else a[i]=0; }: If we enter a new file (FNR==1), then check all entries in the array a. If the value is still 0, it implies that we did not encounter the key of array a in the previous file, so delete it. Otherwise, reset it to zero to start the next cycle.
• ($2 in a) {a[$2]++}: here we process each line of the file. If the entry is in the array a, increment it. This means that all values which are not common will still have a value 0, others a value higher.
• END { for(i in a) if (a[i]!=0) print i }: at the end of all the processing, print whatever is left.