Comments on: Perl and Multiple Line Ending Characters

By: The Cafes » My Next Book

The Cafes » My Next Book — Fri, 09 Feb 2007 19:59:44 +0000

[...] With a little luck, the book should be on store shelves sometime this summer. I’ve already posted a number of questions that arose while writing it. I’m going to be posting a lot more over the next couple of months. I also plan to post many small excerpts from the book for your perusal and comment. I hope you’ll help out by commenting on, caviling, and correcting the draft pieces I’ll be posting here. [...]

By: Elliotte Rusty Harold

Elliotte Rusty Harold — Thu, 11 Jan 2007 18:29:51 +0000

That is indeed the answer, at least for non-one-liners. Thanks. For everyone else's reference the trick is to add use PerlIO::eol; right after the shebang line:

#!/usr/bin/perl
use PerlIO::eol;
#...

Then, after you open the file, use the binmode command to convert all separators to linefeeds like so

open (FILE, $ARGV[0]) or die("Can't open $ARGV[0]\n");
binmode FILE, ":raw:eol(LF)";

Most people will probably need to install PerlIO::eol from CPAN first, since it does not seem to be included with most Perl distributions by default.

By: Peter

Peter — Wed, 10 Jan 2007 15:40:34 +0000

Looks like http://search.cpan.org/~audreyt/PerlIO-eol-0.14/eol.pm would sort this
out. Pity modules aren’t very one liner friendly.

By: Elliotte Rusty Harold

Elliotte Rusty Harold — Tue, 09 Jan 2007 20:46:04 +0000

You probably never noticed the problem because you didn’t have Mac files in the mix. Windows files have carriage returns and line feeds. Thus the lines would still be split correctly, even if there was an extra \r at the end of each line. Mac text files have only carriage returns at the end of each line, so the entire file gets slurped up and treated as a single line.

By: J Donald

J Donald — Tue, 09 Jan 2007 20:45:03 +0000

I cheat - I run the file through dos2unix and unix2dos when going back and forth between Mac/Unix and Windows. I don’t know whether this would be of any help, or whether you need to preserve the line endings as-is.

By: James Orenchak

James Orenchak — Tue, 09 Jan 2007 20:00:11 +0000

After reading this, I don’t understand why I’ve never experienced any problems with this! I’ve written Perl programs for configuring various systems, checking the configurations of systems and for slicing and dicing data without having to cope with Access and Excel. I’ve gone back and forth between Solaris, MS Windows and Linux systems.

Fred Mallot suggests at http://searchopensource.techtarget.com/tip/0,289483,sid39_gci881400,00.html to solve the problem you’re having using something like this, which converts “whatever” to single newlines:
********************************************
perl -i.bak -pe ’s/[

]+$/
/;’ filenames-here

********************************************
Frederick Ducatella from the University of Edinburgh suggests at http://www.inf.ed.ac.uk/teaching/courses/dme/html/perllist.html something that I’ve often used without giving it too much thought:

When you read data in from a file, you will read line by line. So the variable $in will contain an end-of-line character. When you process input you will usually want to get rid of this character. You can do this with the following command:
chomp($in);

If the end-of-line character is always removed when data is read in, I don’t think you’ll have the problem you’ve described, but I’m not sure. Could that be why I never ran across this problem?

By: John Cowan

John Cowan — Mon, 08 Jan 2007 03:36:55 +0000

Things are worse than you think. There’s no way using -0 to express \r\n, though Windows Perl will translate \r\n to just \n on input (though not so Cygwin Perl). Similarly, Mac Classic Perl interprets \n as meaning CR rather than LF, unlike all other Perls (except maybe on EBCDIC mainframes, I don’t know about those), though Mac OS X Perl is Unix Perl and doesn’t do that.

Your best hope is to use one of the various text filters that adjusts line endings.