A number of people have commented that they’ve never experienced a filename with `\n`, or other weird characters, in the wild. On the contrary, I’ve seen this many many times, to the point that I’m defensive about how I write scripts, to compensate.

In my case, these were files created by some other program that contained a bug, where the filename was accidentally set to something like the file contents, say. These other programs were often written by researchers, rather than engineers, who (reasonably so) cared more about their research, than functional correctness. They were also not incentivised to clean-up the dodgy files, if they fix the bug — a big if! — or may not have even be equipped to do so.

You could argue that this is an edge case, but shit happens. Often.

You shouldn't parse the output of ls(1) (wooledge.org) 149 points by tosh on Dec 31, 2021 | past | 148 comments

https://news.ycombinator.com/item?id=29747034

Parsing the output of ls is an antipattern (wooledge.org) 2 points by goranmoomin on Sept 6, 2021 | past

https://news.ycombinator.com/item?id=28435532

Why you shouldn't parse the output of ls(1) (wooledge.org) 2 points by O1111OOO on Dec 24, 2018 | past | 2 comments

https://news.ycombinator.com/item?id=18753980

Parsing ls (wooledge.org) 3 points by tambourine_man on Jan 31, 2016 | past

https://news.ycombinator.com/item?id=11007601

Why you shouldn't parse the output of ls(1) (wooledge.org) 2 points by rosser on Jan 20, 2015 | past

https://news.ycombinator.com/item?id=8914841

Why you shouldn't parse the output of ls (wooledge.org) 31 points by dgellow on July 6, 2014 | past | 18 comments

https://news.ycombinator.com/item?id=7994720

I have been writing shell scripts for over twenty years and don't think I have ever encountered a file name with a newline. I have hit file names starting with a dash - though being interpreted as flags at least a few times. Every command should support the -- path separation feature a lot of newer stuff like git does.

I think that if an attacker can control filenames fully, this is the least of your worries. Otherwise this article could've just as well been titled "why you shouldn't create filenames with newlines", but that makes too much sense.

Has anyone actually found a good use for filenames containing newlines?

This is why I won’t use bash for anything remotely complicated. It’s just too full of edge cases. Any general application language is better JavaScript, F#, heck even Python.

That's why I kind of like PowerShell, object-oriented instead of string-oriented

This is why PowerShell is so great. Commands return objects with a structure so you can do what you want with the output from Get-Childitem safely.

People don't consider adversarial inputs when building scripts, which is one of the main reasons why I started a company to build an incident response platform in 2018.

I was handling security incidents on a large scale with tools that attackers would purposely circumvent with special filenames/inputs. Instead of fighting against the grain, I decided to build a platform with a heavy focus on correctness. Any deviation from the data specification would stick out as a sore thumb and immediately detect anti-forensics and other tricks.

Today, I have a vast compendium of anti-forensics tricks, such as commands not getting written to history, files that cannot be deleted or copied, and much more.

Suffice to say that if you are parsing the output of any tool, you are vulnerable to a whole slew of adversarial techniques.

Use find(1) with -print0

I hung around #bash@freenode (before libera) for a few years and picked up so many bashisms that I now can't look at any script without finding some nitpick. Especially scripts made by enterprise organisations to install proprietary software.

unless, of course, you use (gnu)

  ls -D

and the application for which it was specifically earmarked. I am consistently amazed at how hard it is to mess dired up.

Simple solution: if you ever run into this, blog about it and mark it as a freak occurrence, because it is. Don't ever in real work worry about stupid edge cases like this or you're wasting your employer's money. I'd reprimand an engineer for spending more than an hour on this, it serves nobody unless you're working on extremely widely used library code.

Seriously though, has anyone encountered a file with a new line in its name? I have never encountered this "problem" in the wild.

The more realistic reason is that ls is very slow in directories with a lot of files compared to find.

No one makes files with new line characters so I think I'll be fine

Can’t wait for the iOS 18 feature where it summarizes the page for you with a tap

I disagree, the whole purpose of UNIX is to feed output of one command into another command.

All these file name issues all come from one source, Microsoft. POSIX should be updated to forbid file names with any of these characters:

* space

* NULL

* New Line (as others said, I never ran across a file with a \n in it)