効率的なゲノムファイル（FASTQ、SAM/BAM/CRAM、VCF、GVF、FASTA、PHYLIP、23andMe）の圧縮器 Genozip

2021 7/20 DocumentaitonのURL修正

2021 10/9 コマンド追記

2021 11/9 追記

　ゲノムデータのための汎用的で機能が充実した圧縮ソフトウェアであるGenozipを紹介する。Genozipは、汎用性（一般的なゲノムファイル形式をすべてサポート）、高圧縮率、高速性、機能性、拡張性の5つのコア機能を提供することで、ゲノム圧縮のための汎用ソフトウェアおよび開発フレームワークとして設計されている。

　Genozipは、FASTQ、SAM/BAM/CRAM、VCF、GVF、FASTA、PHYLIP、23andMeフォーマットなど、ゲノム研究で広く使われているゲノムデータフォーマットに対応した高性能圧縮を提供する。テスト結果は、Genozipが高速で、ファイルがすでに圧縮されている場合でも、大幅に改善された圧縮率を達成していることを示している。

　さらに、Genozipは、ファイルフォーマットに特化したセグメンタやデータタイプに特化したコーデックからGenozipフレームワークを分離して設計されている。これにより、Genozipは、研究者が将来的に追加のファイルフォーマットや、ファイル内のデータタイプやフィールドのための新しいコーデックのための圧縮を実装できる汎用圧縮プラットフォームになることを目指している。これにより、最終的には、ユーザーコミュニティによるこれらのアルゴリズムの認知度と採用率が向上し、この分野でのさらなる技術革新が加速することを期待している。

　GenozipはC言語で書かれている。コードはオープンソースで、GitHub (https://github.com/divonlan/genozip)で公開されている。このパッケージは非商用利用の場合は無料である。DockerHub上のDockerコンテナとして、またcondaパッケージマネージャを通じて配布されている。GenozipはLinux、Mac、Windowsでテストされている。

Documentaiton

https://genozip.readthedocs.io

• Genozipはゲノムファイル用の圧縮機でFASTQ、SAM/BAM/CRAM、VCF/BCF、FASTA、GVF、Phylip、23andMeファイルを圧縮するように最適化されているが、ゲノムファイルだけでなく、あらゆるファイルを圧縮できる。すでに .gz .bz2 .xz で圧縮されている場合も圧縮できる。
• 圧縮率は圧縮されるデータに依存する。通常、.bam を圧縮する場合は 1.5～3 倍、.fastq.gz ファイル (つまり既に圧縮されているファイルを圧縮する場合) では 2～5 倍、GT データのみを含む非圧縮の高サンプルカウント .vcf ファイルを圧縮する場合は最大 200 倍の圧縮率が期待できる。
• 圧縮はロスレスで、解凍されたファイルは元のファイルと100%同じ。厳密なロスレス化の例外は --optimize オプションを使用した場合。
• 元のファイルが BGZF で圧縮されていた場合、genounzip は解凍時に BGZF でファイルを再圧縮する(--plain が指定されていない時)。しかし、使用されているライブラリが異なるために、全く同じ BGZF 圧縮はできないことがある。

マニュアルより

2021 7/3 

Genozip v12のリリース。圧縮・解析機能の段階的な向上（RELEASE NOTES参照）に加えて、2つの大きな機能が追加された。１つ目はDual coordinate VCFのサポート。Dual coordinate VCF（link）は、例えばGRCh37とGRCh38のように、2つの座標系の座標を同時に含むVCFファイル。２つ目は、 kraken2を使ったBAMファイルのspeciesフィルタリング機能。バクテリアのリードを直接特定することで、ヒトゲノムデータからバクテリアの汚染をフィルタリングすることができる（詳細）。この機能はBAMファイル（FASTQだけではない）で動作するので、解析のどの時点でも使用することができる。

11/9 

new benchmarks of Genozip

https://genozip.readthedocs.io/benchmarks.html

インストール

ソースからのビルド推奨。（12.08はsegmentation errrorを起こしたので12.07をソースからビルド）

#Form github
git clone https://github.com/divonlan/genozip
make
requires: gcc or clang, make

#conda、ここでは高速なmambaを使う
mamba install -c conda-forge genozip -y

> genozip

$ genozip

Compress genomics files. Genozip can compress any file, but is optimally designed to compress the following file types:

   VCF/BCF, SAM/BAM/CRAM, FASTQ, FASTA, GVF and 23andMe

Usage: genozip [options]... [files or urls]...

One or more file names or URLs may be given, or if omitted, standard input is used instead

Supported input file types, as recognized by their listed filename extension(s): 

   FASTA:   fasta, fa, faa, ffn, fnn, fna (possibly .gz .bgz .bz2 .xz)

   FASTQ:   fastq, fq (possibly .gz .bgz .bz2 .xz)

   SAM:     sam (possibly .gz .bgz .bz2 .xz)

   BAM:     bam

   CRAM:    cram

   VCF:     vcf (possibly .gz .bgz .bz2 .xz)

   BCF:     bcf (possibly .gz .bgz)

   GVF:     gvf (possibly .gz .bgz .bz2 .xz)

   23andMe: genome*Full*.txt (possibly zip)

   Generic: any other file (possibly .gz .bgz .bz2 .xz)

Note: for comressing .bcf, .cram or .xz files requires bcftools, samtools or xz, respectively, to be installed, as does using --index

Examples: genozip sample.bam

          genozip sample.R1.fq.gz sample.R2.fq.gz --pair --reference hg19.ref.genozip -o sample.genozip          genozip --optimize -password 12345 ftp://ftp.ncbi.nlm.nih.gov/file2.vcf.gz

See also: genounzip genocat genols

Actions - use at most one of these actions:

   -d --decompress   Same as running genounzip. For more details, run: genounzip --help

   -l --list         Same as running genols. For more details, run: genols --help

   -h --help         <topic> Show this help page. Optional <topic> can be:

                     dev   - list of developer options

                     input - list of possible arguments of --input

   -L --license      Show the license terms and conditions for this product

   -V --version      Display version number

Flags:

   -i --input        <data-type>. data-type is one of the supported input file types listed above, examples: bam vcf.gz fq.xz. See "genozip --help=input" for full list of accepted file types

                     This flag should be used when redirecting input data with a < or |, or if the input file type cannot be determined by its file name

   -f --force        Force overwrite of the output file, or force writing .genozip data to standard output

   -^ --replace      Replace the source file with the result file, rather than leaving it unchanged

   -o --output       <output-filename>. This option can also be used to bind multiple input files into a single genozip file. The files can be later unbound with 'genounzip --unbind'. To bind files, they must be of the same type (VCF, SAM etc) and if they

                     are VCF files, they must contain the same samples. genozip takes advantage of similarities between the input files so that the bound file is usually smaller than the combined size of individually compressed files

      --best         Best compression, but slower than --fast mode. This is the default mode of genozip - this flag has no additional effect.

   -F --fast         Fast compression, but lower compression ratio than --best. Files compressed with this option also uncompress faster. Compressing with this option also consumes less memory.

   -p --password     <password>. Password-protected - encrypted with 256-bit AES

   -m --md5          Calculate the MD5 digest of the original textual file (vcf, sam...) instead of Adler32. The MD5 is also viewable with genols.

                     Note: for compressed files, e.g. myfile.vcf.gz or myfile.bam, the MD5 calculated is that of the original, uncompressed textual file - myfile.vcf or myfile.sam respectively.

   -I --input-size   <file size in bytes> genozip configures its internal data structures to optimize execution speed based on the file size. When redirecting the input file with < or |, genozip cannot determine its size, and this might result in slower

                     execution. This problem can be overcome by using this flag to inform genozip of the file size

   -q --quiet        Don't show the progress indicator or warnings

   -Q --noisy        The --quiet is turned on by default when outputting to the terminal. --noisy stops the suppression of warnings

   -t --test         After compressing normally, decompresss in memory (i.e. without writing the decompressed file to disk) - comparing the MD5 of the resulting textual (vcf, sam) decompressed file to that of the original textual file. This option also

                     activates --md5

   -@ --threads      <number>. Specify the maximum number of threads. By default, genozip uses all the threads it needs to maximize usage of all available cores

   -B --vblock       <number between 1 and 2048>. Set the maximum size of data (in megabytes) of the textual input (VCF, SAM, FASTQ etc) data that a thread processes at any given time. By default, Genozip sets this value dynamically based on the

                     characateristics of the file, and it is reported in --show-stats. Smaller values will result in faster subsetting with --regions and --grep, while larger values will result in better compression. Note that memory consumption of both

                     genozip and genounzip is linear with the vblock value used for compression

   -e --reference    <filename>.ref.genozip Use a reference file - this is a FASTA file genozipped with the --make-reference option. The same reference needs to be provided to genounzip or genocat.

                     While genozip is capabale of compressing without a reference, in the following cases providing a reference may result in better compression:

                     1. FASTQ files

                     2. SAM/BAM files

                     3. VCF files with significant REFALT content (see "% of zip" in --show-stats)

   -E --REFERENCE    <filename>.ref.genozip Similar to --reference, except genozip copies the reference (or part of it) to the output file, so there is no need to specify --reference in genounzip and genocat.

                     Note on using with --password: the copy of the reference file stored in the compressed file is never encrypted

   --make-reference  Compresss a FASTA file to be used as a reference in --reference or --REFERENCE. Ignored for non-FASTA files

   -w --show-stats   Show the internal structure of a genozip file and the associated compression stats

   -W --SHOW-STATS   Show more detailed stats

   --register        Register (or re-register) a non-commericial license to use genozip

FASTQ-specific options (ignored for other file types):

   -2 --pair         Compress pairs of paired-end FASTQ files, resulting in compression ratios better than compressing the files individually. When using this option, every two consecutive files on the file list should be paired-end FASTQ files with an

                     identical number of reads and consistent file names, and --reference or --REFERENCE must be specified. The resulting genozip file is a bound file. To display interleaved, use genocat --interleaved, and to unbind the genozip file back

                     to its original FASTQ files, use genounzip --unbind.

FASTA-specific options (ignored for other file types):

   --multifasta      All contigs in the FASTA file are variations of a the same contig (i.e. they are somewhat similar to each other). Genozip uses this information to improve the compression.

Optimizing:

   -9 --optimize     Modify the file in ways that are likely insignificant for analytical purposes, but significantly improve compression and somewhat improve the speed of genocat --regions. --optimize activates all these optimizations, or they can be

                     activated individually. These optimizations are:

   VCF optimizations: 

   --optimize-sort   - INFO subfields are sorted alphabetically.               Example: AN=21;AC=3 -> AC=3;AN=21

   --optimize-PL     - PL data: Phred values of over 60 are changed to 60.     Example: '0,18,270' -> '0,18,60'

   --optimize-GL     - GL data: Numbers are rounded to 2 significant digits.   Example: '-2.61618,-0.447624,-0.193264' -> '-2.6,-0.45,-0.19'

   --optimize-GP     - GP data: Numbers are rounded to 2 significant digits, as with GL.

   --optimize-VQSLOD - VQSLOD data: Number is rounded to 2 significant digits. Example: '-4.19494' -> '-4.2'

   SAM optimizations: 

   --optimize-QUAL   - The QUAL quality field and the secondary U2 quality field (if it exists), are modified to group quality scores into a smaller number of bins:

                       Quality scores of 2-9 are changed to 6; 10-19->15 ; 20-24->22 ; 25-29->27 ..... 85-89->87 ; 90-92->91 ; 93 unchanged

                       This assumes a standard Sanger format of Phred quality scores 0->93 encoded in ASCII 33->126

                       Note: this follows Illumina's quality bins for values up to Phred 39, and extends with additional similar bins for values of 40 and above common in some non-Illumina technologies: https://sapac.illumina.com/content/dam/illumina

                     -marketing/documents/products/technotes/technote_understanding_quality_scores.pdf

                       Example: 'LSVIHINKHK' -> 'IIIIFIIIFI'

   --optimize-ZM     - ZM:B:s data: negative Ion Torrent flow signal values are changed to zero, and positives are rounded to the nearest 10.

                       Example: '-20,212,427' -> '0,210,430'

   FASTQ optimizations: 

   --optimize-DESC   - Replaces the description line with '@filename:read_number'

                       Example: '@A00488:61:HMLGNDSXX:4:1101:1561:1000 2:N:0:CTGAAGCT+ATAGAGGC' -> '@sample.fq.gz:100' (100 is the read sequential number within this fastq file)

   --optimize-QUAL   - The quality data is optimized as described for SAM above

   GVF optimizations: 

   --optimize-sort   - Attributes are sorted alphabetically.                   Example: Notes=hi;ID=rs12 -> ID=rs12;Notes=hi

   --optimize-Vf     - Variant_freq data: Number is rounded to 2 significant digits. Example: '0.006351' -> '0.0064'

                     Note: due to these data modifications, files compressed with --optimize are NOT identical to the original file after decompression. For this reason, it is not possible to use this option in combination with --test or --md5

genozip is available for free for non-commercial use and some other limited use cases. See 'genozip -L for details'. Commercial use requires a commercial license

Citing: Lan, D., et al. Bioinformatics, Volume 36, Issue 13, July 2020, Pages 4091-4092

Bug reports and feature requests: bugs@genozip.com

Commercial license inquiries: sales@genozip.com

Requests for support for compression of additional public or proprietary genomic file formats: sales@genozip.com

THIS SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR

ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

> genocat

$ genocat

Print original genomic file(s) previously compressed with genozip

Usage: genocat [options]... [files]...

One or more file names must be given

See also: genozip genounzip genols

Reference-file related options:

   -e --reference    <filename>.ref.genozip Load a reference file prior to decompressing. Required for files compressed with --reference

                     When no non-reference file is specified, display the reference data itself. Typically used in combination with --regions

   -E --REFERENCE    <filename>.ref.genozip with no non-reference file specified. Display the reverse complement of the reference data itself. Typically used in combination with --regions

   --show-reference  Show the name and MD5 of the reference file that needs to be provided to uncompress this file

Subsetting options (options resulting in modified display of the data):

   --downsample      <rate> Show only one in every <rate> lines (or reads in the case of FASTQ). Other subsetting options, if any, will be applied to the surviving lines only.

   --interleaved     For FASTQ data compressed with --pair: Show every pair of paired-end FASTQ files with their reads interleaved: first one read of the first file, then a read from the second file, then the next read from the first file and so on.

   -r --regions      [^]chr|chr:pos|pos|chr:from-to|chr:from-|chr:-to|from-to|from-|-to|from+len[,...]

   VCF SAM FASTA     Show one or more regions of the file. Examples:

   GVF 23andMe                 genocat myfile.vcf.genozip -r22:1000-2000  (Positions 1000 to 2000 on contig 22)

   ref                         genocat myfile.sam.genozip -r22:1000+151   (151 bases, starting pos 1000, on contig 22)

                               genocat myfile.vcf.genozip -r-2000,2500-   (Two ranges on all contigs)

                               genocat myfile.sam.genozip -rchr21,chr22   (Contigs chr21 and chr22 in their entirety)

                               genocat myfile.vcf.genozip -r^MT,Y         (All contigs, excluding MT and Y)

                               genocat myfile.vcf.genozip -r^-1000        (All contigs, excluding positions up to 1000)

                               genocat myfile.fa.genozip  -rchrM          (Contig chrM)

                     Note: genozip files are indexed automatically during compression. There is no separate indexing step or separate index file

                     Note: Indels are considered part of a region if their start position is

                     Note: Multiple -r arguments may be specified - this is equivalent to chaining their regions with a comma separator in a single argument

                     Note: For FASTA files, only whole-contig regions are possible

   -s --samples      [^]sample[,...]

   VCF               Show a subset of samples (individuals). Examples:

                               genocat myfile.vcf.genozip -s HG00255,HG00256    (show two samples)

                               genocat myfile.vcf.genozip -s ^HG00255,HG00256   (show all samples except these two)

                     Note: This does not change the INFO data (including the AC and AN tags)

                     Note: sample names are case-sensitive

                     Note: Multiple -s arguments may be specified - this is equivalent to chaining their samples with a comma separator in a single argument

   -g --grep         <string> Show only records in which <string> is a case-sensitive substring of the description

   FASTQ FASTA

   --list-chroms     List the names of the chromosomes (or contigs) included in the file

   VCF SAM FASTA GVF 23andMe 

   -G --drop-genotypes Output the data without the samples and FORMAT column

   VCF

   -H --no-header    Don't output the header lines

   -1 --header-one   VCF: Output only the last line on the header (the line with the field and sample names)

   VCF FASTA         FASTA: Output the sequence name up to the first space or tab

   --header-only     Output only the header lines

   --GT-only         For samples, output only genotype (GT) data, dropping the other subfields

   VCF

   --sequential      Output in sequential format - each sequence in a single line

   FASTA

Translation options (options resulting convertion of the data from one format to another):

      --bam          (SAM and BAM only) Output as BAM.

                     Note: this option is implicit if --output specifies a filename ending with .bam

      --sam          (SAM and BAM only) Output as SAM. This option is the default in genocat on SAM and BAM data.

      --fastq        (SAM and BAM only) Output as FASTQ

                     The alignments are outputed as FASTQ reads in the order they appear in the SAM/BAM file. Alignments with FLAG 16 (reverse complimented) have their SEQ reverse complimented and their QUAL reversed. Alignments with FLAG 4 (unmapped) or

                     256 (secondary) are dropped. Alignments with FLAG 64 (or 128) (the first (or last) segment in the template) have a '1' (or '2') added after the read name. Usually, if the original order of the SAM/BAM file has not been tampered with,

                     this would result in a valid interleaved FASTQ file.

                     Note: this option is implicit if --output specifies a filename ending with .fq[.gz] or .fastq[.gz]

      --bcf          (VCF only) Output as BCF, using bcftools

                     Note: bcftools needs to be installed for this option to work

      --phylip       (FASTA only) Output a Multi-FASTA in Phylip format. All sequences must be the same length

      --fasta        (Phylip only) Output as Multi-FASTA

      --vcf          (23andMe only) Output as VCF. --vcf must be used in combination with --reference to specify the reference file as listed in the header of the 23andMe file (usually this is GRCh37)

                     Note: INDEL genotypes ('DD', 'DI', 'II') as well as uncalled sites ('--') are discarded

General options:

   -o --output       <output -filename>. Output to this filename instead of stdout

   -z --bgzf         <level>. Compress the output to the BGZF format (.gz extension) using libdeflate, at the compression level specified in the argument. Argument specifies the compression level from 0 (no compression) to 12 (best, yet slowest,

                     compression). If you're not sure what value to choose, 6 is a popular option.

                     Note: by default, genocat's output is not compressed. Unlike genounzip, genocat makes no attempt to reconstruct the compression level of the original file

   -p --password     Provide password to access file(s) that were compressed with --password

   -@ --threads      Specify the maximum number of threads. By default, genozip uses all the threads it needs to maximize usage of all available cores

   -x --index        Create an index file alongside the decompressed file, when combined with --output. The index file is created using 'samtools index' for SAM/BAM files, 'samtools faidx' for FASTA/FASTQ files and 'bcftools index' for VCF files. Other

                     file formats cannot be indexed

   -q --quiet        Don't show warnings

   -Q --noisy        The --quiet is turned on by default when outputting to the terminal. --noisy stops the suppression of warnings

   -w --show-stats   Show the internal structure of a genozip file and the associated compression stats

   -W --SHOW-STATS   Show more detailed stats

   -h --help         <topic> Show this help page. Optional <topic> can be:

                     dev - list of developer options

   -L --license      Show the license terms and conditions for this product

   -V --version      Display version number

Tip regarding using genozip files in a pipeline:

   Option 1: For tools that support input redirection - use a regular pipe. Example:

             genocat myfile.vcf.genozip | bcftools view -

   Option 2: For tools that don't support input redirection - use a named pipe. Example:

             mkfifo mypipe.vcf

             genocat myfile.vcf.genozip > mypipe.vcf &

             othertool mypipe.vcf

Citing: Lan, D., et al. Bioinformatics, Volume 36, Issue 13, July 2020, Pages 4091-4092

Bug reports and feature requests: bugs@genozip.com

Commercial license inquiries: sales@genozip.com

Requests for support for compression of additional public or proprietary genomic file formats: sales@genozip.com

THIS SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR

ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

> genounzip

$ genounzip

Uncompress genomic files previously compressed with genozip

Usage: genounzip [options]... [files]...

One or more file names must be given

Examples: genounzip file1.vcf.genozip file2.sam.genozip

          genounzip file.vcf.genozip --output file.vcf.gz

          genounzip bound.vcf.genozip --unbind

See also: genozip genocat genols

Options:

   -x --index        Create an index file alongside the decompressed file. The index file is created using 'samtools index' for SAM/BAM files, 'samtools faidx' for FASTA/FASTQ files and 'bcftools index' for VCF files. Other file formats cannot be indexed

   -c --stdout       Send output to standard output instead of a file

   -z --bgzf         <level>. Compress the output to the BGZF format (.gz extension) using libdeflate, at the compression level specified in the argument. Argument specifies the compression level from 0 (no compression) to 12 (best, yet slowest,

                     compression). If you're not sure what value to choose, 6 is a popular option.

                     Note: by default, absent this option, genozip will attempt to re-create the same BGZF compression as in the original file. Whether genozip succeeds in re-creating the exact same BGZF compression ratio depends on the compression library

                     used by the application that generated the original file.

      --no-PG        (SAM and BAM only) When converting a file from SAM to BAM or vice versa, Genozip normally adds a @PG line in the header. With this option, it doesn't

   -f --force        Force overwrite of the output file

   -^ --replace      Replace the source file with the result file, rather than leaving it unchanged

   -u --unbind[=prefix] Split a bound file back to its original components. If the '--unbind=prefix' form is used, a prefix is added to each file component. A prefix may include a directory.

   -o --output       <output-filename>. Output to this filename instead of the default one

   -e --reference    <filename>.ref.genozip Load a reference file prior to decompressing. Required for files compressed with --reference

   -p --password     <password>. Provide password to access file(s) that were compressed with --password

   -m --md5          Show the digest of the decompressed file - MD5 if the file was compressed with --md5 or --test and Adler32 if not.

                     Note: for compressed files, e.g. myfile.vcf.gz, the digest calculated is that of the original, uncompressed file. 

   -q --quiet        Don't show the progress indicator or warnings

   -Q --noisy        The --quiet is turned on by default when outputting to the terminal. --noisy stops the suppression of warnings

   -t --test         Decompress in memory (i.e. without writing the decompressed file to disk) and use the digest (MD5 or Adler32) to verify that the resulting decompressed file is identical to the original file.

   -@ --threads      <number>. Specify the maximum number of threads. By default, genozip uses all the threads it needs to maximize usage of all available cores

   -w --show-stats   Show the internal structure of a genozip file and the associated compression stats

   -W --SHOW-STATS   Show more detailed stats

   -h --help         <topic> Show this help page. Optional <topic> can be:

                     dev - list of developer options

   -L --license      Show the license terms and conditions for this product

   -V --version      Display version number

Translation options (options resulting convertion of the data from one format to another):

      --bam          (SAM and BAM only) Output as BAM.

                     Note: this option is implicit if --output specifies a filename ending with .bam

      --sam          (SAM and BAM only) Output as SAM. This option is the default in genocat on SAM and BAM data.

      --fastq        (SAM and BAM only) Output as FASTQ

                     The alignments are outputed as FASTQ reads in the order they appear in the SAM/BAM file. Alignments with FLAG 16 (reverse complimented) have their SEQ reverse complimented and their QUAL reversed. Alignments with FLAG 4 (unmapped) or

                     256 (secondary) are dropped. Alignments with FLAG 64 (or 128) (the first (or last) segment in the template) have a '1' (or '2') added after the read name. Usually, if the original order of the SAM/BAM file has not been tampered with,

                     this would result in a valid interleaved FASTQ file.

                     Note: this option is implicit if --output specifies a filename ending with .fq[.gz] or .fastq[.gz]

      --bcf          (VCF only) Output as BCF, using bcftools

                     Note: bcftools needs to be installed for this option to work

      --phylip       (FASTA only) Output a Multi-FASTA in Phylip format. All sequences must be the same length

      --fasta        (Phylip only) Output as Multi-FASTA

      --vcf          (23andMe only) Output as VCF. --vcf must be used in combination with --reference to specify the reference file as listed in the header of the 23andMe file (usually this is GRCh37)

                     Note: INDEL genotypes ('DD', 'DI', 'II') as well as uncalled sites ('--') are discarded

Citing: Lan, D., et al. Bioinformatics, Volume 36, Issue 13, July 2020, Pages 4091-4092

Bug reports and feature requests: bugs@genozip.com

Commercial license inquiries: sales@genozip.com

Requests for support for compression of additional public or proprietary genomic file formats: sales@genozip.com

THIS SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR

ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

> genols 

$ genols -h

View metadata of genomic files previously compressed with genozip

Usage: genols [options]... [files or directories]...

One or more file or directory names may be given, or if omitted, genols runs on the current directory

See also: genozip genounzip genocat

Options:

   -u --unbind       Show the components of bound files. This option is implied when running genols on a single file

   -b --bytes        Show sizes in bytes

   -q --quiet        Don't show warnings

   -h --help         Show this help page

   -L --license      Show the license terms and conditions for this product

   -V --version      Display version number

Citing: Lan, D., et al. Bioinformatics, Volume 36, Issue 13, July 2020, Pages 4091-4092

Bug reports and feature requests: bugs@genozip.com

Commercial license inquiries: sales@genozip.com

Requests for support for compression of additional public or proprietary genomic file formats: sales@genozip.com

THIS SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR

ANYONE DISTRIBUTING THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY, WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

実行方法

初回は端末上で非商用の利用であるかの確認と、ユーザー登録のためのメッセージが表示される。最後に登録するのメッセージが出る。登録しないと使用できない。

 fastqを圧縮する。--replaceをつけると元のファイルは消える。

genozip input.fastq --replace

#fastq.gzをさらに圧縮（データによっては2/3くらいになる）
genozip input.fq.gz

#複数ファイル（個別に圧縮される）
genozip fastq_dir/*.fq.gz

• --replace      Replace the source file with the result file, rather than leaving it unchanged 
• -F --fast Fast compression,   but lower compression ratio than --best. Files compressed with this option also uncompress faster. Compressing with this option also consumes less memory.
• -p  --password <password>.  Password-protected - encrypted with 256-bit AES
• -@ --threads <number>.  Specify the maximum number of threads. By default, genozip uses all the threads it needs to maximize usage of all available cores

input.fastq.genozipができる。gzip圧縮ファイルも.gzが消えてinput.fastq.genozipとして出力される。

解凍する。--replaceをつけると元のファイルは消える。gzip圧縮ファイルをさらに圧縮した場合も解凍するとraw  fastqに解凍される。

genounzip input.fastq.genozip --replace

• -c --stdout    Send output to standard output instead of a file
• --replace      Replace the source file with the result file, rather than leaving it unchanged

リファレンスゲノム（fasta）とそのリファレンスに対応するペアエンドfastqを圧縮する。 単体で行うより圧縮効率が高くなる（よりファイルサイズが小さくなる）。

#まずはリファレンスを圧縮(少し時間がかかるがリファレンスは１回圧縮すれば使い回せる)
genozip --make-reference genome.fasta
#=> genome.ref.genozipができる

#次にペアエンドfastqを圧縮、paired-end fastqとgenome.ref.genozipを指定する。
genozip -2 pair_1.fq pair_2.fq -e genome.ref.genozip

• --make-reference   Compresss a FASTA file to be used as a reference in --reference or --REFERENCE. Ignored for non-FASTA files

• -e --reference <filename>.  ref.genozip Use a reference file - this is a FASTA file genozipped with the --make-reference option. The same reference needs to be provided to genounzip or genocat.
  While genozip is capabale of compressing without a reference, in the following cases providing a reference may result in better compression:
  1. FASTQ files
  2. SAM/BAM files
  3. VCF files with significant REFALT content (see "% of zip" in --show-stats)

• -E --REFERENCE <filename>.  ref.genozip Similar to --reference, except genozip copies the reference (or part of it) to the output file, so there is no need to specify --reference in genounzip and genocat.
  Note on using with --password: the copy of the reference file stored in the compressed file is never encrypted

• -2 --pair   Compress pairs of paired-end FASTQ files, resulting in compression ratios better than compressing the files individually. When using this option, every two consecutive files on the file list should be paired-end FASTQ files with an identical number of reads and consistent file names, and --reference or --REFERENCE must be specified. The resulting genozip file is a bound file. To display interleaved, use genocat --interleaved, and to unbind the genozip file back to its original FASTQ files, use genounzip --unbind. 

pair_1+2.fastq.genozipが出力される。

Genozip圧縮ファイルのメタデータを確認する。

genols pair_1+2.fastq.genozip

2021 7/3

Dual coordinate VCFの作成。

genozip --chain mychain.chain.genozip mydata.vcf

fastqの例を書きましたが、他にもたくさんのファイルタイプやファイルタイプの組み合わせに対応しています。サポートされているファイルタイプの完全なリストは 'genozip --help=input'で確認できます。間違って2回紹介してしまいましたが、残しておきます。

引用

Genozip - A Universal Extensible Genomic Data Compressor
Divon Lan, Ray Tobler, Yassine Souilmi, Bastien Llamas Author Notes
Bioinformatics, Published: 15 February 2021

追記

2022/09

Genozip 14 - advances in compression of BAM and CRAM files
Divon Lan,  Bastien Llamas

bioRxiv, posted September 14, 2022

関連

自分用

カレント以下のfq.gzを全て圧縮

find . -name "*.fq.gz" -type f -print0 | while read -r -d '' file; do
genozip "$file" "--replace"; done