Processing DIA data with DIA-NN on Linux
Kustatscher lab tutorial
This tutorial provides a complete pipeline for processing DIA data using DIA-NN 2.2 on a Linux server. The Linux version of DIA-NN operates through the command line. To simplify the execution of the necessary commands, which can be quite complex, we have organised them into a series of numbered bash and R scripts that together form a streamlined data processing pipeline. The workflow consists of the following steps:
Transfer raw mass spectrometry data to the processing server
Convert vendor-specific raw data to the mzML format (
0_mzML_convert.sh; ZenoTOF only)Inspect total ion chromatograms (
1_total_ion_chromatograms.R)Generate a spectral library from a FASTA file using deep learning (
2_predict_library.sh)Optimise DIA-NN search parameters (
3_diann_parameter_search.sh)Process the data with DIA-NN (
4_diann_process.sh)Calculate protein quantities from precursor data (
5_protein_quant.R)Generate a QC report (
6_qc_report.qmd)
The pipeline is designed to be run using Positron, which offers an integrated environment to execute bash and R commands via Remote SSH. All scripts are available in our raw-data-processing GitHub repository and come pre-loaded on our Linux server. For more information on DIA-NN and its parameters, see the official DIA-NN documentation.
To follow this tutorial you should meet the following requirements:
A basic understanding of the Linux command line and the Positron IDE (Positron homepage, lab guide). The tutorial uses Positron for all steps, as this contains all the necessary tools (terminal, R console, remote ssh) and provides the same interface for all users regardless of their OS. Alternatively, you could connect to the server and run the pipeline using RStudio Server or any terminal.
You have access to our Linux server, Ernie, and are a member the docker and sharedfolder_user_group user groups (lab wiki).
You have mounted both the Proteomics core datastore (read-only) and the Kustatscher lab datastore on Ernie (lab wiki).
DIA-NN is installed for all Ernie users, but R packages need to be installed by each user separately. To install all R packages required for this pipeline, run
pak::pak(c("arrow", "data.table", "ggplot2", "scales", "DT", "iq", "rawrr", "mzR")), followed byrawrr::installRawrrExe().
1 Set up the project on the Linux server
As a first step, we create a folder on the server that contains all the raw MS data and all the processing scripts.
1.1 Set up a project folder
- Open Positron and connect to the Ernie Linux server using the Remote Explorer (lab wiki)
- Select New folder > Empty Project and give the folder an informative name (do not initialize a Git repository).
Positron will automatically set the working directory to this new folder. All commands in the tutorial are run from within the project folder.
1.2 Start a tmux session (optional)
This pipeline takes hours to complete, so we recommend running it inside a tmux session. Tmux keeps your terminal session alive on the server even if you close your laptop or lose your connection — you can reconnect later and pick up exactly where you left off. We recommend running every step in this tutorial inside the same tmux session, only closing it at the very end.
In the TERMINAL tab, start a new session:
tmux new -s myanalysisThe -s myanalysis gives the session a name, so you can find it again later (even from a different PC). A green bar at the bottom of the terminal confirms you are inside a tmux session.
Useful shortcuts:
| Command / shortcut | Action |
|---|---|
Ctrl+b then d |
Detach from the session (leaves it running in the background) |
tmux attach -t myanalysis |
Reattach to the session (from any terminal) |
Ctrl+b then [ |
Enter copy mode to scroll back through output. Press q to exit. |
tmux kill-session -t myanalysis |
End the session permanently |
See the lab wiki for more tmux commands.
1.3 Transfer MS Data to the project folder
Our raw MS data are stored on the proteomics core’s datastore for permanent storage. We need to copy the relevant raw files to our project folder for processing.
In the TERMINAL tab, define the proteomics core folder as the “source” path and inspect its contents:
SRC=/home/$USER/datastore_proteomicscore/1_MS_Raw_Files/<instrument>/<year>/<month>/<dataset>
ls -lh $SRCNext, copy the .raw files to project folder, excluding blank and cleaning runs:
rsync -vh --progress --exclude='*Blank*' --exclude='*CLEAN*' $SRC/*.raw $PWD/| Argument | Meaning |
|---|---|
$SRC/*.raw |
Source. All .raw files in the $SRC folder. For ZenoTOF data, use $SRC/*.wiff* instead, which will copy .wiff, .wiff2 and .wiffscan. |
$PWD/ |
Destination. $PWD is a shortcut for the current working directory |
--exclude |
Skips files whose names contain the specified pattern. |
The transfer takes around 10 seconds per GB. Use ls -lh to inspect the downloaded data. Should you need to delete any files, such as .raw files belonging to an unrelated project, the safest option is to right-click & delete them in Positron’s file explorer.
1.4 Transfer pipeline scripts
Finally, copy all pipeline scripts into your project folder. This keeps everything self-contained. You can edit the scripts for your specific dataset, and the folder serves as a permanent record of exactly which scripts were used.
cp /srv/sharedfolder/raw-data-processing/diann-pipeline/* $PWD/2 Convert MS files to mzML format
Skip this step for .raw files (Lumos, Exploris, Astral). It is only necessary for .wiff files (ZenoTOF).
The 0_mzML_convert.sh script converts vendor files to mzML format using msconvert. It also performs peak-picking and verifies that all output files are complete.
./0_mzML_convert.sh --threads 6 --file_type .wiff --slack <member-ID>| Argument | Description |
|---|---|
--threads |
Number of parallel processing threads (1–88 on Ernie). Each thread spins up one Docker container for one input file, so there is no benefit in using more threads than files. Book threads on Ernie’s booking calendar. |
--file_type |
Type of raw files to convert: .raw, .wiff, or .wiff2. |
--slack |
(Optional) Your Slack member ID (looks like U0XXXXXXXXX). If provided, the script will send you a Slack notification when it finishes. |
Outcome: One .mzML file per input file, plus a log file (0_mzML_convert.log). A summary is printed in the terminal reporting the number of files converted and whether all mzML files are complete. Expect 20–50 minutes for 4 GB raw files.
3 Inspect total ion chromatograms
Before running a search, it is useful to inspect the total ion chromatograms (TICs) to check for anomalies such as failed injections or low-intensity runs, and if necessary exclude problematic files from the downstream search.
The 1_total_ion_chromatograms.R script reads all MS files in your project folder, extracts the MS1 TIC data and plots the chromatograms to a PDF file.
Open the script by clicking on it in Positron’s file explorer. In the configuration section at the top, set file_type to either "raw" or "mzML" to match the files in your project folder. Then click the Source button (▷) to run the whole script.
Outcome: A single PDF file (TICs.pdf) in the project folder, showing one chromatogram per injection.
4 Generate a Spectral Library
A spectral library is required for DIA-NN searches. This can be generated from a FASTA file using deep learning.
4.1 Download FASTA sequences from UniProt
UniProt is the best source for protein sequence data. For many species they have pre-compiled FASTA files that contain exactly one protein sequence per gene. For example, you can download the human one here. However, the following approach allows more flexibility and will work for most species and sequence variants.
- Visit UniProt’s Proteomes page and locate your species.
- Select the Entry link (e.g. UP000005640 for human).
- Click the Number of entries link to access the detailed protein list (e.g. around 80,000 for human).
- For well-annotated proteomes such as human, S. cerevisiae or E. coli, restrict this list to curated, non-redundant sequences by clicking on Reviewed (Swiss-Prot) in the top left corner (around 20,000 for human). Omit this step for poorly annotated proteomes, such as chicken or Cryptococcus. For such species we are forced to work with computationally generated, redundant sequences from the Unreviewed (trEMBL) collection.
- Click Download to open the download menu. By default, UniProt offers to download FASTA (canonical) sequences, which means gene-level information. This is useful for method development, but for many biological questions we are interested in isoform sequences as well. For such experiments change this to FASTA (canonical & isoform).
- Set Compressed to No.
- Click Generate URL for API. Copy the API URL for the streaming endpoint. This is a link to the file you are about to download.
- Switch back to the TERMINAL tab in Positron and download the FASTA file using the command line (replace the URL with the one you copied for your species):
# Download file from the API URL and save it as hs_SP_iso_7Mar26.fasta
curl "https://rest.uniprot.org/uniprotkb/stream?format=fasta&includeIsoform=true&query=%28proteome%3AUP000005640%29+AND+%28reviewed%3Atrue%29" -o hs_SP_iso_7Mar26.fastaAn informative file name helps to remember which types of sequences are in the FASTA file, e.g. hs = homo sapiens, SP = SwissProt, iso = includes isoforms, 7Mar26 = download date.
4.2 Predict a spectral library from the FASTA file
The 2_predict_library.sh script generates a spectral library from a FASTA file using DIA-NN’s built-in deep learning predictor. The script automatically detects the single .fasta file in the project folder — make sure there is exactly one present.
./2_predict_library.sh --threads 20 --slack <member-ID>| Argument | Description |
|---|---|
--threads |
Number of parallel processing threads. Book threads on Ernie’s booking calendar. |
--slack |
(Optional) Your Slack member ID. |
Outcome: A spectral library named <FASTA>_OxAc.predicted.speclib, where the _OxAc suffix indicates it was generated with Ox(M) and Ac(N-term) as variable modifications. Also produces a log file (2_predict_library.log). Expect around 5 minutes with 20 threads for a human SwissProt database with isoforms, producing a 1.5 GB library.
To predict a spectral library using the DIA-NN GUI: Load FASTA file, click FASTA digest for library-free search, which automatically enables Deep learning-based spectra and Generate spectral library. Enable Ox(M) and Ac(N-term) as modifications, specify output names, select number of threads, disable MBR and Quantities matrices and click Run.
5 Optimise DIA-NN search parameters
DIA-NN requires accurate settings for key search parameters, including mass accuracy (MS1 and MS2) and scan window. By default, DIA-NN estimates these from the first processed file, but testing multiple files systematically can improve performance.
The 3_diann_parameter_search.sh script runs DIA-NN on a random subset of files in your project folder, determines optimal values per file, and reports overall recommended settings.
./3_diann_parameter_search.sh --label LF --threads 20 --n_files 6 --slack <member-ID>| Argument | Description |
|---|---|
--label |
Labelling strategy: LF, SILAC, or pulse-SILAC. LF = label-free; SILAC and pulse-SILAC only differ in normalisation strategy. |
--threads |
Number of parallel processing threads. |
--n_files |
Number of random files to include in parameter optimisation. As a guideline, use all files for small experiments (< 15 samples) and half of the files for larger experiments. |
--slack |
(Optional) Your Slack member ID. |
Assumptions: The script assumes that the project folder contains all the MS files, either in .raw or .mzML format, as well as a single predicted spectral library and the matching FASTA file.
Outcome: Recommended values for --window, --mass_acc, and --ms1_mass_acc are printed in the terminal and saved to a log file. Typical runtime is around 45 minutes for six 4 GB files using 20 threads.
Load a random subset of MS files, a predicted spectral library and matching FASTA file. Enable Ox(M) and Ac(N-term) as modifications, and Unrelated runs. Specify output names, select number of threads, disable Generate Spectral Library, MBR, Quantities matrices and click Run.
6 Perform the DIA-NN search
After identifying the optimal search parameters, run the actual DIA-NN data processing with the 4_diann_process.sh script. This script implements a two-pass workflow (also known as match-between-runs, MBR), but provides outputs that allow retrospective consideration of both single-pass and two-pass strategies.
./4_diann_process.sh --label LF --threads 20 --window 7 --mass_acc 11 --ms1_mass_acc 8 --slack <member-ID>| Argument | Description |
|---|---|
--label |
Labelling strategy: LF, SILAC, or pulse-SILAC. LF = label-free; SILAC and pulse-SILAC only differ in normalisation strategy. |
--threads |
Number of parallel processing threads to use (1–88 on Ernie). |
--window |
Scan window radius (as determined by 3_diann_parameter_search.sh) |
--mass_acc |
MS2 mass accuracy |
--ms1_mass_acc |
MS1 mass accuracy |
--slack |
(Optional) Your Slack member ID. |
Assumptions: The script assumes that the project folder contains all the MS files, either in .raw or .mzML format, as well as a single predicted spectral library and the matching FASTA file.
Outcome: A first-pass report (the result of the library-free, single-pass strategy) and a second-pass report (report.parquet, produced by reanalysing the data using the refined spectral library generated during the first pass). The script also outputs the refined spectral library, log files and stats files. After processing, this script automatically removes raw MS files and other intermediate files to free up disk space (the original data remain on the proteomics core datastore). Typical runtime is around 1 hour for six 4 GB files using 20 threads.
Load all MS files, a predicted spectral library and matching FASTA file. Enable Ox(M) and Ac(N-term) as modifications. Select number of threads. Specify values for Mass accuracy, MS1 accuracy and Scan window. Set output directories and click run.
7 Protein Quantification in R
DIA-NN’s primary output is a precursor-level report that needs to be processed into protein-level quantities. The 5_protein_quant.R script performs this step for both label-free and SILAC experiments. It also applies FDR and quality filters.
7.1 Prepare a sample mapping file
Before running the script, create a CSV file called sample_mapping.csv in your project folder. This file maps each DIA-NN run name (e.g. E240731_01_Exploris_ARB_IN_180_R1_SS48_Strap) to shorter, human-readable annotations. The sample mapping is essential for grouping runs into conditions and replicates, and for producing readable plots in the QC report.
The file must contain these four columns:
| Column | Description |
|---|---|
Run |
The run name exactly as it appears in the DIA-NN report |
Sample |
A short, unique alias for the run |
Condition |
Experimental condition |
Replicate |
Replicate identifier |
For example, for the tutorial dataset:
| Run | Sample | Condition | Replicate |
|---|---|---|---|
| E240731_01_Exploris_ARB_IN_180_R1_SS48_Strap | R1_SS48 | SS48 | R1 |
| E240731_03_Exploris_ARB_IN_180_R2_SS48_Strap | R2_SS48 | SS48 | R2 |
| E240731_05_Exploris_ARB_IN_180_R3_SS48_Strap | R3_SS48 | SS48 | R3 |
| E240731_07_Exploris_ARB_IN_180_R1_SS6_Strap | R1_SS6 | SS6 | R1 |
| E240731_09_Exploris_ARB_IN_180_R2_SS6_Strap | R2_SS6 | SS6 | R2 |
| E240731_11_Exploris_ARB_IN_180_R3_SS6_Strap | R3_SS6 | SS6 | R3 |
To avoid copy-paste errors when entering long run names, generate a template file with the Run column pre-filled from the DIA-NN report itself. In the R console, run:
runs <- unique(read_parquet("report.parquet", col_select = "Run"))[[1]]
fwrite(data.table(Run = runs, Sample = "", Condition = "", Replicate = ""), "sample_mapping.csv")You now have three options for filling in the remaining columns:
- Edit in Positron. Right-click
sample_mapping.csvin the file explorer and select Open With → Text Editor. Type the values directly and save. - Edit in Excel. Right-click the file and select Download to save it to your local PC. Open in Excel, fill in the columns, save as CSV, then drag the file back into Positron’s file explorer to overwrite the version on Ernie.
- Populate programmatically. If your run names follow a consistent pattern, skip the manual step and fill the columns using regular expressions in R:
sample_mapping <- data.table(Run = runs)
sample_mapping[, Sample := sub(".*(R._SS.+)_Strap", "\\1", Run)]
sample_mapping[, Condition := sub(".*_(SS.+)_Strap", "\\1", Run)]
sample_mapping[, Replicate := sub(".*_(R.)_.*", "\\1", Run)]
fwrite(sample_mapping, "sample_mapping.csv")Adjust the regular expressions to match your run naming pattern.
7.2 Configure and run the script
Open 5_protein_quant.R in Positron and edit the configuration section at the top. Once configured, click the Source button (▷) to run the script.
The script produces two files: the protein-level quantities for each sample (protein_quant.csv.gz) and the filtered precursor-level report (report_filtered.parquet).
Key columns in protein_quant.csv.gz
| Column | Description |
|---|---|
Run |
Name of the injection |
Sample, Condition, Replicate |
Sample annotations from sample_mapping.csv |
Protein_group |
Protein UniProt IDs (key identifier) |
Genes, Protein_names |
Gene and protein annotation |
LFQ |
MaxLFQ intensity (label-free only) |
LFQ_H, LFQ_L |
MaxLFQ intensity per SILAC channel (SILAC only) |
log2_HL_ratio |
log2 heavy/light SILAC ratio (SILAC only) |
N_ratios |
Number of precursor ratios contributing to SILAC ratio (SILAC only) |
Intensity |
Sum of normalised precursor intensities |
Intensity_H, Intensity_L |
Per-channel intensity (SILAC only) |
8 QC Report
The 6_qc_report.qmd is a Quarto document that generates an HTML quality control report from the outputs of the previous step. It provides an overview of the dataset, including identification numbers, intensity distributions, data completeness, missed cleavages, reproducibility (CV curves), and PCA plots.
Open 6_qc_report.qmd in Positron and edit the configuration cell at the top. Once configured, click the Preview button in Positron to render the report. This will produce a self-contained HTML file (6_qc_report.html) in the project folder.
For example, the tutorial dataset produces this QC report.
9 Archive and downstream analysis
At this point, the project folder on Ernie contains everything produced by the pipeline: the FASTA file, spectral library, DIA-NN reports, protein quantities, QC report, and log files recording all processing parameters. The following steps download the key output files for downstream analysis, archive the full project folder for permanent record-keeping, and clean up Ernie.
9.1 Download the key outputs for local analysis
The main outputs of the pipeline (protein_quant.csv.gz and 6_qc_report.html) are the starting point for all downstream analysis. They are small enough to analyse on your local PC. In Positron’s file explorer on Ernie, right-click each file and select Download to save them into a dedicated local folder.
Downstream analysis can then happen in that local folder, ideally tracked via a private GitHub repository under the kustatscher-lab organisation.
9.2 Archive the project folder to the group datastore
The full project folder is archived to the Kustatscher lab datastore. The Projects folder on the datastore mirrors the structure of the lab’s Notion workspace, and each dataset gets its own subfolder named by experiment and date (e.g. Exp01-E240731).
From the TERMINAL tab, move the project folder to the datastore using rsync with --remove-source-files. This copies the files across and then deletes the originals on Ernie:
rsync -av --progress --remove-source-files $PWD/ /home/$USER/datastore_kustatscherlab/Projects/<project_name>/<Exp##-ID>/Adjust the destination path (<project_name>/<Exp##-ID>/…) to match the Notion structure. The destination folder will be created by rsync if it does not yet exist.
This archive is the permanent record of the processing step. Everything needed to reproduce the result - pipeline scripts, parameters, logs, and reports - is captured.
9.3 Close the tmux session
tmux kill-session -t myanalysis