Merge branch 'master' of github.com:minoda-lab/universc into dev

test/shared/icell8-test/72618_KU812_L001_R1_001.fastq.gz filter=lfs diff=lfs merge=lfs -text

test/shared/icell8-test/72618_KU812_L001_R2_001.fastq.gz filter=lfs diff=lfs merge=lfs -text

test/shared/icell8-test/72618_KU812_L002_R1_001.fastq.gz filter=lfs diff=lfs merge=lfs -text

test/shared/cellranger-tiny-fastq/3.0.0/tinygex_S1_L001_I1_001.fastq.gz filter=lfs diff=lfs merge=lfs -text

test/shared/cellranger-tiny-fastq/3.0.0/tinygex_S1_L001_R1_001.fastq.gz filter=lfs diff=lfs merge=lfs -text

test/shared/cellranger-tiny-fastq/3.0.0/tinygex_S1_L001_R2_001.fastq.gz filter=lfs diff=lfs merge=lfs -text

test/shared/cellranger-tiny-fastq/3.0.0/tinygex_S1_L002_I1_001.fastq.gz filter=lfs diff=lfs merge=lfs -text

test/shared/cellranger-tiny-fastq/3.0.0/tinygex_S1_L002_R1_001.fastq.gz filter=lfs diff=lfs merge=lfs -text

test/shared/cellranger-tiny-fastq/3.0.0/tinygex_S1_L002_R2_001.fastq.gz filter=lfs diff=lfs merge=lfs -text

- add support for new technologies:

  -  5′ scRNA technologies (e.g., Smart-Seq3, ICELL8 5′ , STRT-Seq)

  -  full support for combinatorial indexing (e.g., BD Rhapsody, Microwell-Seq, SCI-RNA-Seq, SPLiT-Seq, SureCell/ddSEQ)

  -  full support for dual indexes (e.g., inDrops v3, SCI-RNA-Seq, Smart-Seq3)

  -  full support for dual indexes (e.g., Fluidigm C1, ICELL8 full-length, inDrops v3, SCI-RNA-Seq, Smart-Seq3)

  -  non-UMI technologies (e.g., ICELL8-v2, Quartz-Seq, RamDA-Seq, Smart-Seq2, STRT-Seq)

- add support for barcodes longer than 16 bp

- support custom whitelists from mixed samples with a low fraction of reads

- update test cases for renaming files and compressed inputs

- bug fixes for computing summary statistics

### 1.0.3

We provide the following preset configurations for convenience based on published data and configurations used by other pipelines 

(e.g, DropSeqPipe and Kallisto/Bustools). To add further support for other technologies or troubleshoot problems, please submit an Issue

to the GitHub repository: [TomKellyGenetics/universc](https://github.com/TomKellyGenetics/universc/issues)

to the GitHub repository: [minoda-lab/universc](https://github.com/minoda-lab/universc/issues)

as described in [Bug Reports](#Issues) below.

Some changes to the Cell Ranger install are required to run other technologies. Therefore we provide settings for 10x Genomics

-  ICELL8

    -  ICELL8 version 2 (11 bp barcode, No UMI): icell8-non-umi, icell8-v2

    -  ICELL8 version 3 (11 bp barcode, 14 bp UMI): icell8 or custom

    -  ICELL8 5′ scRNA with TCR OR kit (10bp barcode, 8 bp UMI): icell8-5-prime

    -  ICELL8 5′ scRNA with TCR OR kit (10bp barcode, NO bp UMI): icell8-5-prime

    -  ICELL8 full-length scRNA with Smart-Seq (16 bp barcode, No UMI): icell8-full-length

-  inDrops

    -  inDrops version 1 (19 bp barcode, 6 bp UMI): indrops-v1, 1cellbio-v1

#### Single and dual indexed technologies

Where needed the cell barcode can be detected in the index I1 or I2 file.

Single indexes are supported for STRT-Seq, Quartz-Seq, and RamDA-Seq.

Dual indexes are supported for inDrops-v3, SCI-RNA-Seq, scifi-seq, and Smart-Seq.

Single indexes are supported for STRT-Seq and Quartz-Seq.

Dual indexes are supported for Fluidigm C1, ICELL8 full-length, 

inDrops-v3, RamDA-Seq, SCI-RNA-Seq, scifi-seq, and Smart-Seq.

Combinatorial indexing technologies have linkers between barcodes removed

automatically to match the barcode whitelist.

#### Demultiplexing for dual-indexing

For dual-indexed technologies such as inDrops-v3, Sci-Seq, SmartSeq3 it is advised to use "bcl2fastq"

before calling UniverSC:

For dual-indexed technologies such as Fluidigm C1, inDrops-v3, Sci-Seq,

SmartSeq3 it is advised to use "bcl2fastq" before calling UniverSC:

```

   /usr/local/bin/bcl2fastq  -v --runfolder-dir "/path/to/illumina/bcls"  --output-dir "./Data/Intensities/BaseCalls"\

```

Please adjust the lengths for `--use-bases-mask` accordingly for read 1, index 1 (i7), index 2 (i5), and read 2.

Ensure that `--create-fastq-for-index-read` is used where possible. If a sequencing facility has demultiplexed

the samples for you without this, UniverSC will attempt to extract index sequences from FASTQ headers in read 1.

Ensure that `--create-fastq-for-index-read` is used where possible.

Using `--no-lane-splitting` is optional as UniverSC can process an arbirtary number of lanes.

There is no need to specify index sequences in the same sheet for cell barcodes, using "NNNNNNNN" will match all

samples and the cell barcodes will be distinguished by the single-cell processing pipeline. Index sequences should

only be used to demultiplex samples and replicates (not cells).

#### Missing index sequences

If a sequencing facility has demultiplexed the samples for you without this,

UniverSC will attempt to extract index sequences from FASTQ headers in read 1.

If index sequences are not stored in the file headers and samples have already

been demultiplexed, a dummy index file of the same number of reads as R1 and R2

will be required. As a workaroudn, you can generate this by copying the R1 and R2

files and replacing the sequences with the first barcode in the relevant whitelist.

For example:

```

index1="TAAGGCGA"

index2="AAGGAGTA"

# create new files

cp R1_file.fastq I1_file.fastq

cp R2_file.fastq I2_file.fastq

# replace sequences

sed -i "2~4s/^.*$/${index1}/g" I1_file.fastq

sed -i "2~4s/^.*$/${index2}/g" I2_file.fastq

# replace quality scores

sed -i "4~4s/^.*$/IIIIIIII/g" I1_file.fastq I2_file.fastq

```

This results in a new "sample index" for each demultiplexed sample.

To combine demultiplexed sampls for dual indexed techniques use the following:

```

# for fastq files

cat Sample1_R1_file.fastq Sample2_R1_file.fastq Sample3_R1_file.fastq > Combined_R1_file.fastq

cat Sample1_R2_file.fastq Sample2_R2_file.fastq Sample3_R2_file.fastq > Combined_R2_file.fastq

cat Sample1_I1_file.fastq Sample2_I1_file.fastq Sample3_I1_file.fastq > Combined_I1_file.fastq

cat Sample1_I2_file.fastq Sample2_I2_file.fastq Sample3_I2_file.fastq > Combined_I2_file.fastq

# for compressed files (not need to uncompress)

cat Sample1_R1_file.fastq.gz Sample2_R1_file.fastq.gz Sample3_R1_file.fastq.gz > Combined_R1_file.fastq.gz

cat Sample1_R2_file.fastq.gz Sample2_R2_file.fastq.gz Sample3_R2_file.fastq.gz > Combined_R2_file.fastq.gz

cat Sample1_I1_file.fastq.gz Sample2_I1_file.fastq.gz Sample3_I1_file.fastq.gz > Combined_I1_file.fastq.gz

cat Sample1_I2_file.fastq.gz Sample2_I2_file.fastq.gz Sample3_I2_file.fastq.gz > Combined_I2_file.fastq.gz

```

As this needs to done on a case-by-case basis it has not been implemented by the UniverSC core functions.

We provide this workaround for using published data and data already processed by sequencing facilities.

Please contact the maintainers or file an issue on GitHub if you are having problems with this case.

#### Custom inputs

Custom inputs are also supported by giving the name "custom" and length of barcode and UMI separated by a "_" character.

#### Reporting issues

To add further support for other technologies or troubleshoot problems, please submit an Issue 

to the GitHub repository: https://github.com/TomKellyGenetics/universc/issues

to the GitHub repository: https://github.com/minoda-lab/universc/issues

Please submit [issues](https://github.com/minoda-lab/universc/issues) on GitHub to report

problems or suggest features. [Pull requests](https://github.com/minoda-lab/universc/pulls)

```

cd $HOME/Downloads

git clone https://github.com/TomKellyGenetics/universc.git

git clone https://github.com/minoda-lab/universc.git

cd universc

```

```

cd $/HOME/my_git_repo

git submodule add https://github.com/TomKellyGenetics/universc.git

git submodule add https://github.com/minoda-lab/universc.git

bash universc/launch_universc.sh

```

The code is available here:

[https://github.com/TomKellyGenetics/cellranger/releases](https://github.com/TomKellyGenetics/cellranger/releases)

[https://github.com/minoda-lab/cellranger/releases](https://github.com/minoda-lab/cellranger/releases)

We also provide Docker images for Cell Ranger versions 2.0.2, 2.1.0, 2.1.1, and 3.0.2:

[https://github.com/TomKellyGenetics/cellranger_clean/packages](https://github.com/TomKellyGenetics/cellranger_clean/packages)

[https://github.com/minoda-lab/cellranger_clean/packages](https://github.com/minoda-lab/cellranger_clean/packages)

[https://hub.docker.com/r/tomkellygenetics/cellranger_clean/tags](https://hub.docker.com/r/tomkellygenetics/cellranger_clean/tags)

These have been pre-installed in the Docker image described above.

A full example of installation is available in the [GitHub repository](https://github.com/TomKellyGenetics/cellranger)

A full example of installation is available in the [GitHub repository](https://github.com/minoda-lab/cellranger)

and on [DockerHub](https://hub.docker.com/r/tomkellygenetics/cellranger_clean/dockerfile).

- Python 2.7.13

```

cd $HOME/Downloads

git clone https://github.com/TomKellyGenetics/universc.git

git clone https://github.com/minoda-lab/universc.git

cd universc

bash launch_universc.sh

```

```

cd $HOME/Downloads

git clone https://github.com/TomKellyGenetics/universc.git

git clone https://github.com/minoda-lab/universc.git

make

make install prefix=$HOME/local

```

are still compatible.

```

git clone https://github.com/TomKellyGenetics/universc.git

git clone https://github.com/minoda-lab/universc.git

docker build -t universc:latest .  

```

                                  Drop-Seq (12 bp barcode, 8 bp UMI): dropseq

                                  ICELL8 version 2 (11 bp barcode, No UMI): icell8-non-umi, icell8-v2

                                  ICELL8 version 3 (11 bp barcode, 14 bp UMI): icell8 or custom

                                  ICELL8 5′ scRNA with TCR OR kit (10bp barcode, 8 bp UMI): icell8-5-prime

                                  ICELL8 5′ scRNA with TCR OR kit (10bp barcode, NO bp UMI): icell8-5-prime

                                  ICELL8 full-length scRNA with Smart-Seq (16 bp barcode, No UMI): icell8-full-length

                                  inDrops version 1 (19 bp barcode, 6 bp UMI): indrops-v1, 1cellbio-v1

                                  inDrops version 2 (19 bp barcode, 6 bp UMI): indrops-v2, 1cellbio-v2

   index: 1

 - name: "RIKEN Center for Sustainable Resource Sciences, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama, Kanagawa 230-0045, Japan"

   index: 2

date: "Thursday 06 May 2021"

date: "Wednesday 12 May 2021"

output:

  prettydoc::html_pretty:

       theme: cayman

![GitHub issues](https://img.shields.io/github/issues/minoda-lab/universc)

![GitHub pull requests](https://img.shields.io/github/issues-pr/minoda-lab/universc)

[![HitCount](http://hits.dwyl.com/minoda-lab/universc.svg)](http://hits.dwyl.com/minoda-lab/universc)

[![HitCount](http://hits.dwyl.com/tomkellygenetics/universc.svg)](http://hits.dwyl.com/tomkellygenetics/universc)

[![GitHub Views](http://hits.dwyl.com/minoda-lab/universc.svg)](http://hits.dwyl.com/minoda-lab/universc)

[![GitHub Views](http://hits.dwyl.com/tomkellygenetics/universc.svg)](http://hits.dwyl.com/tomkellygenetics/universc)

![GitHub search hit counter](https://img.shields.io/github/search/minoda-lab/universc/master)

![GitHub forks](https://img.shields.io/github/forks/minoda-lab/universc?style=social)

![GitHub Repo stars](https://img.shields.io/github/stars/minoda-lab/universc?style=social)

We provide the following preset configurations for convenience based on published data and configurations used by other pipelines 

(e.g, DropSeqPipe and Kallisto/Bustools). To add further support for other technologies or troubleshoot problems, please submit an Issue

to the GitHub repository: [TomKellyGenetics/universc](https://github.com/TomKellyGenetics/universc/issues)

to the GitHub repository: [minoda-lab/universc](https://github.com/minoda-lab/universc/issues)

as described in [Bug Reports](#Issues) below.

Some changes to the Cell Ranger install are required to run other technologies. Therefore we provide settings for 10x Genomics

-  ICELL8

    -  ICELL8 version 2 (11 bp barcode, No UMI): icell8-non-umi, icell8-v2

    -  ICELL8 version 3 (11 bp barcode, 14 bp UMI): icell8 or custom

    -  ICELL8 5′ scRNA with TCR OR kit (10bp barcode, 8 bp UMI): icell8-5-prime

    -  ICELL8 5′ scRNA with TCR OR kit (10bp barcode, NO bp UMI): icell8-5-prime

    -  ICELL8 full-length scRNA with Smart-Seq (16 bp barcode, No UMI): icell8-full-length

-  inDrops

    -  inDrops version 1 (19 bp barcode, 6 bp UMI): indrops-v1, 1cellbio-v1

#### Single and dual indexed technologies

Where needed the cell barcode can be detected in the index I1 or I2 file.

Single indexes are supported for STRT-Seq, Quartz-Seq, and RamDA-Seq.

Dual indexes are supported for inDrops-v3, SCI-RNA-Seq, scifi-seq, and Smart-Seq.

Single indexes are supported for STRT-Seq and Quartz-Seq.

Dual indexes are supported for Fluidigm C1, ICELL8 full-length, 

inDrops-v3, RamDA-Seq, SCI-RNA-Seq, scifi-seq, and Smart-Seq.

Combinatorial indexing technologies have linkers between barcodes removed

automatically to match the barcode whitelist.

#### Demultiplexing for dual-indexing

For dual-indexed technologies such as inDrops-v3, Sci-Seq, SmartSeq3 it is advised to use "bcl2fastq"

before calling UniverSC:

For dual-indexed technologies such as Fluidigm C1, inDrops-v3, Sci-Seq,

SmartSeq3 it is advised to use "bcl2fastq" before calling UniverSC:

```

   /usr/local/bin/bcl2fastq  -v --runfolder-dir "/path/to/illumina/bcls"  --output-dir "./Data/Intensities/BaseCalls"\

```

Please adjust the lengths for `--use-bases-mask` accordingly for read 1, index 1 (i7), index 2 (i5), and read 2.

Ensure that `--create-fastq-for-index-read` is used where possible. If a sequencing facility has demultiplexed

the samples for you without this, UniverSC will attempt to extract index sequences from FASTQ headers in read 1.

Ensure that `--create-fastq-for-index-read` is used where possible.

Using `--no-lane-splitting` is optional as UniverSC can process an arbirtary number of lanes.

There is no need to specify index sequences in the same sheet for cell barcodes, using "NNNNNNNN" will match all

samples and the cell barcodes will be distinguished by the single-cell processing pipeline. Index sequences should

only be used to demultiplex samples and replicates (not cells).

#### Missing index sequences

If a sequencing facility has demultiplexed the samples for you without this,

UniverSC will attempt to extract index sequences from FASTQ headers in read 1.

If index sequences are not stored in the file headers and samples have already

been demultiplexed, a dummy index file of the same number of reads as R1 and R2

will be required. As a workaroudn, you can generate this by copying the R1 and R2

files and replacing the sequences with the first barcode in the relevant whitelist.

For example:

```

index1="TAAGGCGA"

index2="AAGGAGTA"

# create new files

cp R1_file.fastq I1_file.fastq

cp R2_file.fastq I2_file.fastq

# replace sequences

sed -i "2~4s/^.*$/${index1}/g" I1_file.fastq

sed -i "2~4s/^.*$/${index2}/g" I2_file.fastq

# replace quality scores

sed -i "4~4s/^.*$/IIIIIIII/g" I1_file.fastq I2_file.fastq

```

This results in a new "sample index" for each demultiplexed sample.

To combine demultiplexed sampls for dual indexed techniques use the following:

```

# for fastq files

cat Sample1_R1_file.fastq Sample2_R1_file.fastq Sample3_R1_file.fastq > Combined_R1_file.fastq

cat Sample1_R2_file.fastq Sample2_R2_file.fastq Sample3_R2_file.fastq > Combined_R2_file.fastq

cat Sample1_I1_file.fastq Sample2_I1_file.fastq Sample3_I1_file.fastq > Combined_I1_file.fastq

cat Sample1_I2_file.fastq Sample2_I2_file.fastq Sample3_I2_file.fastq > Combined_I2_file.fastq

# for compressed files (not need to uncompress)

cat Sample1_R1_file.fastq.gz Sample2_R1_file.fastq.gz Sample3_R1_file.fastq.gz > Combined_R1_file.fastq.gz

cat Sample1_R2_file.fastq.gz Sample2_R2_file.fastq.gz Sample3_R2_file.fastq.gz > Combined_R2_file.fastq.gz

cat Sample1_I1_file.fastq.gz Sample2_I1_file.fastq.gz Sample3_I1_file.fastq.gz > Combined_I1_file.fastq.gz

cat Sample1_I2_file.fastq.gz Sample2_I2_file.fastq.gz Sample3_I2_file.fastq.gz > Combined_I2_file.fastq.gz

```

As this needs to done on a case-by-case basis it has not been implemented by the UniverSC core functions.

We provide this workaround for using published data and data already processed by sequencing facilities.

Please contact the maintainers or file an issue on GitHub if you are having problems with this case.

#### Custom inputs

Custom inputs are also supported by giving the name "custom" and length of barcode and UMI separated by a "_" character.

#### Reporting issues

To add further support for other technologies or troubleshoot problems, please submit an Issue 

to the GitHub repository: https://github.com/TomKellyGenetics/universc/issues

to the GitHub repository: https://github.com/minoda-lab/universc/issues

Please submit [issues](https://github.com/minoda-lab/universc/issues) on GitHub to report

problems or suggest features. [Pull requests](https://github.com/minoda-lab/universc/pulls)

```

cd $HOME/Downloads

git clone https://github.com/TomKellyGenetics/universc.git

git clone https://github.com/minoda-lab/universc.git

cd universc

```

```

cd $/HOME/my_git_repo

git submodule add https://github.com/TomKellyGenetics/universc.git

git submodule add https://github.com/minoda-lab/universc.git

bash universc/launch_universc.sh

```

The code is available here:

[https://github.com/TomKellyGenetics/cellranger/releases](https://github.com/TomKellyGenetics/cellranger/releases)

[https://github.com/minoda-lab/cellranger/releases](https://github.com/minoda-lab/cellranger/releases)

We also provide Docker images for Cell Ranger versions 2.0.2, 2.1.0, 2.1.1, and 3.0.2:

[https://github.com/TomKellyGenetics/cellranger_clean/packages](https://github.com/TomKellyGenetics/cellranger_clean/packages)

[https://github.com/minoda-lab/cellranger_clean/packages](https://github.com/minoda-lab/cellranger_clean/packages)

[https://hub.docker.com/r/tomkellygenetics/cellranger_clean/tags](https://hub.docker.com/r/tomkellygenetics/cellranger_clean/tags)

These have been pre-installed in the Docker image described above.

A full example of installation is available in the [GitHub repository](https://github.com/TomKellyGenetics/cellranger)

A full example of installation is available in the [GitHub repository](https://github.com/minoda-lab/cellranger)

and on [DockerHub](https://hub.docker.com/r/tomkellygenetics/cellranger_clean/dockerfile).

- Python 2.7.13

```

cd $HOME/Downloads

git clone https://github.com/TomKellyGenetics/universc.git

git clone https://github.com/minoda-lab/universc.git

cd universc

bash launch_universc.sh

```

```

cd $HOME/Downloads

git clone https://github.com/TomKellyGenetics/universc.git

git clone https://github.com/minoda-lab/universc.git

make

make install prefix=$HOME/local

```

are still compatible.

```

git clone https://github.com/TomKellyGenetics/universc.git

git clone https://github.com/minoda-lab/universc.git

docker build -t universc:latest .  

```

                                  Drop-Seq (12 bp barcode, 8 bp UMI): dropseq

                                  ICELL8 version 2 (11 bp barcode, No UMI): icell8-non-umi, icell8-v2

                                  ICELL8 version 3 (11 bp barcode, 14 bp UMI): icell8 or custom

                                  ICELL8 5′ scRNA with TCR OR kit (10bp barcode, 8 bp UMI): icell8-5-prime

                                  ICELL8 5′ scRNA with TCR OR kit (10bp barcode, NO bp UMI): icell8-5-prime

                                  ICELL8 full-length scRNA with Smart-Seq (16 bp barcode, No UMI): icell8-full-length

                                  inDrops version 1 (19 bp barcode, 6 bp UMI): indrops-v1, 1cellbio-v1

                                  inDrops version 2 (19 bp barcode, 6 bp UMI): indrops-v2, 1cellbio-v2