Remove repeated lines from a block of text, keeping only the first occurrence of each.
How to Use the Remove Duplicate Lines Tool
Paste your text into the box, with one item per line.
Choose whether comparison should be "Case Sensitive" and whether to "Trim Whitespace Before Comparing".
Click "Remove Duplicates" to see the de-duplicated list and how many lines were removed.
How Duplicates Are Detected
The text is split at every line break. The tool then walks down the list from top to bottom and keeps a record of every line it has already seen. A line is written to the result only if an equal line has not appeared above it. This means:
The original order is kept. Nothing is sorted, so the first occurrence stays in its original position.
Only whole lines are compared. Repeated words inside a line, or a line that merely contains another, are left alone.
Empty lines count as lines too. After the first blank line, later blank lines are treated as duplicates and removed, which also tidies up extra spacing.
The two options only change how lines are compared. The line that is kept is copied exactly as you typed it, with its original spaces and capital letters.
Worked Example: How the Options Change the Result
Suppose you paste these 7 lines: Mumbai, Delhi, mumbai, "Pune " (with a trailing space), Delhi, Pune, Chennai. The number of lines removed depends on the two checkboxes:
Case Sensitive
Trim Whitespace
Lines kept
Removed
On
On
Mumbai, Delhi, mumbai, Pune, Chennai
2
On
Off
Mumbai, Delhi, mumbai, Pune, Pune, Chennai
1
Off
On
Mumbai, Delhi, Pune, Chennai
3
Off
Off
Mumbai, Delhi, Pune, Pune, Chennai
2
With trimming off, "Pune " and "Pune" look identical on screen but are different strings, so both survive. For most lists, keeping trimming on and case sensitivity off gives the cleanest result.
Common Uses and Things to Watch
Cleaning email or mobile number lists merged from several spreadsheets before sending an invite or SMS.
Removing repeated keywords, URLs or product codes copied from reports.
Checking a roll-number or attendance list for entries typed twice: the "Duplicate lines removed" count tells you how many repeats there were.
Formatting differences defeat matching. "98765 43210" and "9876543210", or "Ltd." and "Limited", are different lines. Standardise the format first, for example with the Text Formatter.
If you copy columns from a spreadsheet, only identical whole rows are removed. Copy just the column you want to de-duplicate.
Frequently Asked Questions
It compares each line's text to every line seen so far, and removes any line that repeats an earlier one, keeping only the first occurrence.
When enabled, 'Hello' and 'hello' are treated as different lines. When disabled, they're treated as duplicates regardless of capitalization.
When enabled, leading/trailing spaces are ignored when comparing lines, so ' Hello' and 'Hello' are treated as duplicates, though the original line is kept as-is in the output.
No. Lines stay in their original order and each one is kept at the position where it first appears. If you also want the list in alphabetical order, run the result through the Text Sorter.
An empty line is compared like any other line, so only the first blank line survives and the rest are counted as duplicates. If you need paragraph spacing preserved, de-duplicate a plain one-item-per-line list instead.