Perl6: large gzipped files read line by line

  • A+
Category:Languages

I'm trying to read a gz file line by line in Perl6, however, I'm getting blocked:

  1. How to read gz file line by line in Perl6 however, this method, reading everything into :out uses far too much RAM to be usable except on very small files.

  2. I don't understand how to use Perl6's Compress::Zlib to get everything line by line, although I opened an issue on their github https://github.com/retupmoca/P6-Compress-Zlib/issues/17

  3. I'm trying Perl5's Compress::Zlib to translate this code, which works perfectly in Perl5:

use Compress::Zlib; my $file = "data.txt.gz"; my $gz = gzopen($file, "rb") or die "Error reading $file: $gzerrno";  while ($gz->gzreadline($_) > 0) {     # Process the line read in $_ }  die "Error reading $file: $gzerrno" if $gzerrno != Z_STREAM_END ; $gz->gzclose() ; 

to something like this using Inline::Perl5 in Perl6:

use Compress::Zlib:from<Perl5>; my $file = 'chrMT.1.vcf.gz'; my $gz = Compress::Zlib::new(gzopen($file, 'r'); while ($gz.gzreadline($_) > 0) {   print $_; } $gz.gzclose(); 

but I can't see how to translate this :(

  1. I'm confused by Lib::Archive example https://github.com/frithnanth/perl6-Archive-Libarchive/blob/master/examples/readfile.p6 I don't see how I can get something like item 3 here

  2. There should be something like

for $file.IO.lines(gz) -> $line { or something like that in Perl6, if it exists, I can't find it.

How can I read a large file line by line without reading everything into RAM in Perl6?

 


Update Now tested, which revealed an error, now fixed.

Solution #2

use Compress::Zlib;  my $file   = "data.txt.gz" ; my $handle = try open $file or die "Error reading $file: $!" ; my $zwrap  = zwrap($handle, :gzip) ;  for $zwrap.lines {     .print }  CATCH { default { die "Error reading $file: $_" } }  $handle.close ; 

I've tested this with a small gzipped text file.

I don't know much about gzip etc. but figured this out based on:

  • Knowing P6;

  • Reading Compress::Zlib's README and choosing the zwrap routine;

  • Looking at the module's source code, in particular the signature of the zwrap routine our sub zwrap ($thing, :$zlib, :$deflate, :$gzip);

  • And trial and error, mainly to guess that I needed to pass the :gzip adverb.


Please comment on whether my code works for you. I'm guessing the main thing is whether it's fast enough for the large files you have.

A failed attempt at solution #5

With solution #2 working I would have expected to be able to write just:

use Compress::Zlib ; .print for "data.txt.gz".&zwrap(:gzip).lines ; 

But that fails with:

No such method 'eof' for invocant of type 'IO::Path' 

This is presumably because this module was written before the reorganization of the IO classes.

That led me to @MattOates' IO::Handle like object with .lines ? issue. I note no response and I saw no related repo at https://github.com/MattOates?tab=repositories.

Comment

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: