What the duplicate line remover does
The tool reads your text one line at a time and keeps only one copy of each line. The lines that remain stay in their original order, unless you choose to sort them. A summary shows how many lines went in, how many came out, and which lines were repeated and how many times.
How to remove duplicate lines
- Paste your text in the box, or use Open a text file for a .txt, .csv or .log file.
- Choose your options. The result updates as you type.
- Copy the result, or download it as a text file.
Options
- Ignore case: treats Apple and apple as the same line.
- Trim spaces: ignores spaces and tabs at the start and end of each line, and removes them from the result.
- Remove empty lines too: drops blank lines instead of keeping the first one.
- Keep first or last occurrence: decides which copy of a repeated line stays, and so where it sits in the result.
- Sort result: orders the remaining lines from A to Z or from Z to A, with numbers in natural order.
Why lines that look the same are not removed
- One of them has a space or a tab at the end. Keep Trim spaces switched on.
- They differ in capital letters. Switch on Ignore case.
- One uses a non-breaking space or another invisible character. Trim spaces removes the ones at the edges, but a hidden character in the middle makes the lines different.
Windows, Mac and Linux line endings are all handled, so a file saved on another system works as it is.
Common uses
- Cleaning email lists, keyword lists and lists of URLs or domains.
- Removing repeated entries from log files and exports.
- Preparing a list of IDs or codes for a database query.
- Tidying one column copied from a spreadsheet.
Privacy
Your text is processed in your browser. Nothing is uploaded to a server.
Related tools
To see counts for every line, or only the lines that appear once, use unique lines. Clean up blank lines with remove empty lines, repeated words with remove duplicate words, and order the result with sort lines. You can also reverse lines, shuffle lines or check the total with the line counter. To do it in Excel, Notepad++ or the terminal, read the remove duplicate lines guide.