Great, thank you for summarizing this so neatly. We could even link from the PBMC3k tutorial to this.

Quick answer: yes, there are many situations in which pp.regress_out can do more harm than good. In most cases, I personally don't correct for mitochondrial gene expression, for instance. Quite a few people will agree with that. Whether a certain processing step makes sense or not depends on the data. You should choose with subject knowledge. Because, of that, I found it hard to come up with a best practice tutorial; what came into life as Benchmark with Seurat, remained Seurat's way of defining best practice. Many papers stick to this. But I'm sure that also many Seurat users won't always regress out mitochondrial genes and number of counts per cell.

I'm sure @LuckyMD has a lot to say on this. :)

Hi @grst,

We've been working on a best-practices workflow/tutorial for scRNA-sesq data for the past year. The tutorial is based on scanpy, but it also integrates tools from R. You can take a look at a case study which implements the workflow here. The revised manuscript should be submitted this week as well, so I hope it will be out soon. If you like, I can send it to you, if you pass me your email address.

Regarding regressing out covariates. @falexwolf has already mentioned that this is very dataset dependent. It is also dependent on the downstream analysis. This is especially true for the covariates you suggest. MT gene expression and cell cycle are both biological, rather than technical covariates. Regressing out biological covariates is generally done to isolate particular processes in the data that you are interested in, while losing global structure in the data. This is helpful especially for trajectory inference, but maybe less so for global exploratory analysis with clustering. For example, cell cycle stage can be a major determinant in the difference between two cell types (e.g. stem cells and proliferating cells like transit amplifying cells). Removing this effect, hides the distinction. MT gene expression is also a biological covariate (as well as a technical indicator of cell stress). Higher levels of MT gene expression can indicate increased respiration in e.g., asthmatic conditions.

An additional unwanted effect to regressing out biological covariates is that biological processes are not independent. Thus, regressing out one process, will partially remove effects of other, downstream processes as well.

If you're interested, we can discuss this in more detail. Hope this helps a bit,

Thanks for your comments.
In this case, I will indeed use the function more cautiously in the future.

@LuckyMD, I would be really interested in the manuscript. I'll send you an email regarding that.

Best,
Gregor

Hi @LuckyMD ,
I'm dealing with a large dataset. I just found that pp.regress_out would densify a scipy.sparse._csc.csc_matrix, which might lead to death of Jupyter kernel due to running out of memory. Then I expanded the RAM of my cloud server, and it ran pp.regress_out smoothly.But, the size of the anndata object came to 39 GB from 8GB. That was shocking! I tried scipy.sparse.csr_matrix(adata.X) to recover it to sparse matrix, the kernel died again. I have no idea then.
I had used Seurat long before and the step of regression in Seurat does not change the type of matrix nor the size of the object.
Do you have any idea about how to avoid densifying the matrix?
Thanks for your help~

I’m not sure how regress_out could avoid densifying the matrix. Can you show an example of running Seurat’s regress_out on a sparse matrix that retains a similar amount of zeros in the matrix?