Linux uniq Command

Linux Command Manual

The Linux uniq command is used to check and remove duplicate lines in a text file, typically used in conjunction with the sort command.

uniq can check for duplicate lines in a text file.

Syntax

uniq [-cdu][-f<fields>][-s<char-position>][-w<char-position>][--help][--version][input-file][output-file]

Parameters:

-c or --count: Display the count of occurrences next to each line.
-d or --repeated: Only display duplicate lines.
-f<fields> or --skip-fields=<fields>: Ignore specified fields.
-s<char-position> or --skip-chars=<char-position>: Ignore specified characters.
-u or --unique: Only display lines that appear once.
-w<char-position> or --check-chars=<char-position>: Specify characters to compare.
--help: Display help information.
--version: Display version information.
[input-file]: Specify a sorted text file. If not specified, data is read from standard input.
[output-file]: Specify the output file. If not specified, content is displayed on standard output (terminal).

Examples

In the file testfile, lines 2, 3, 5, 6, 7, and 9 are identical. Use the uniq command to remove duplicate lines with the following command:

uniq testfile

The original content of testfile is:

$ cat testfile      # Original content  
test 30  
test 30  
test 30  
Hello 95  
Hello 95  
Hello 95  
Hello 95  
Linux 85  
Linux 85

After using the uniq command to remove duplicate lines, the output is:

$ uniq testfile     # Content after removing duplicates  
test 30  
Hello 95  
Linux 85

To check the file and remove duplicate lines, displaying the count of occurrences at the beginning of each line, use the following command:

uniq -c testfile

The output is:

$ uniq -c testfile      # Content after removing duplicates  
3 test 30             # The number indicates the line appears 3 times  
4 Hello 95            # The number indicates the line appears 4 times  
2 Linux 85            # The number indicates the line appears 2 times

$ cat testfile1      # Original content 
test 30  
Hello 95  
Linux 85 
test 30  
Hello 95  
Linux 85 
test 30  
Hello 95  
Linux 85

You can use sort:

$ sort testfile1 | uniq
Hello 95  
Linux 85 
test 30

To count the occurrences of each line in the file:

$ sort testfile1 | uniq -c
   3 Hello 95  
   3 Linux 85 
   3 test 30

To find duplicate lines in the file:

$ sort testfile1 | uniq -d
Hello 95  
Linux 85 
test 30

Linux Command Manual

❮ Linux Comm Ed Linux Comm Id ❯