Remove Duplicate Lines

List of lines

Case sensitive

Trim surrounding spaces

Keep blank lines

Result

Unique linesяблоко груша слива
Unique3
Duplicates removed2
Lines before5
Removed, %40%
Comparison modecase insensitive

The order survives

Keeps only the unique lines of a list in their original order, with or without regard to case.

Lists of email addresses, part numbers, keywords and links almost always arrive with repeats. Sorting exposes them but destroys the order, and the order often matters: the first occurrence is usually the current one.

This keeps the first occurrence of each line and drops the rest without disturbing anything else. Case is handled separately: whether 'Apple' and 'apple' are one line or two depends on the job.

What this calculator shows

The list without repeats, in the original order
How many lines there were, how many remain and how many were removed
The share of the original list that was removed

What to keep in mind

The first occurrence is kept; later duplicates are removed.
Leading and trailing whitespace is trimmed by default, or a line with a stray space would count as unique.

FAQs

Is the order preserved?

Yes, the first occurrence of each line stays where it was. No sorting is applied.

Is case taken into account?

Not by default, but it is a toggle. Email addresses generally do not care about case; part numbers and passwords do.

What happens to whitespace?

Leading and trailing spaces are trimmed before comparison by default, or otherwise identical lines would count as different.

What about blank lines?

They are removed by default. If blank lines are part of the structure, turn on the option to keep them.

Worked example

A list with repeats

Input: apple, pear, Apple, plum, pear — case insensitive

Output: 3 unique lines, 2 removed

Note: The capitalised 'Apple' went as a duplicate. With case sensitivity on, four lines would remain — and for a list of part numbers that may well be the right behaviour.

Useful links

All tools in Text tools for word count, character count, and quick cleanup

Remove Duplicate Lines

Result

Related tools

Line Sorter & Deduplicator

Text Cleaner

Find and Replace Tool

Useful links