scAdam model optimization

scAdam model optimization#

Model optimization refers to the process of improving the performance and efficiency of machine learning models. scAdam models support two types of optimization:

  1. Warm start: Warm starting is a technique in machine learning that involves initializing a model with weights from a previously trained model on the same or a similar task. This method allows the training process to begin from a more advantageous point on the loss surface, leveraging prior knowledge to improve efficiency and performance.

  2. Hyperparameter tuning: This involves adjusting the hyperparameters of the model, such as learning rates, batch sizes, and other configuration settings, to optimize performance on a specific dataset.

[1]:

# Python packages
import warnings
warnings.simplefilter('ignore')

import scanpy as sc
import scparadise
import os
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

sc.set_figure_params(dpi = 120)

Recommendations about training dataset#

We recommend the shifted logarithm data normalization method:

sc.pp.normalize_total(adata, target_sum=None)
sc.pp.log1p(adata)

However, you may use another data normalization method, provided that the same method is applied consistently to the test dataset.

The training dataset must contain all genes intended for model training in adata_train.X. We recommend removing genes outside the selected marker-gene set from adata_train.X, as excluding uninformative features may improve model performance and evaluation metrics while reducing training time.

Data preparation#

Here we download and preprocess dataset from cellxgene:

Single cell RNA sequencing of oropharyngeal squamous cell carcinoma https://cellxgene.cziscience.com/collections/3c34e6f1-6827-47dd-8e19-9edcd461893f

[2]:

# Download data from figshare
# You can use wget, or download the file in a browser by copying and pasting the link into a new tab.
link = 'https://datasets.cellxgene.cziscience.com/5ee8e8dd-6f0e-4336-aec8-f476b0b0089b.h5ad'

[3]:

# Load prepared for training anndata object
adata = sc.read_h5ad('5ee8e8dd-6f0e-4336-aec8-f476b0b0089b.h5ad')

[4]:

# Get raw counts from adata.raw
adata = adata.raw.to_adata()

[5]:

# Use gene symbols rather than Ensembl gene IDs (ENSG identifiers).
adata.var.set_index('feature_name', inplace=True)
adata.var_names = adata.var_names.astype('str')
adata.var_names_make_unique()

[6]:

# Select datasets for model training and testing
# It is necessary for all cell types to be present in both the training datasets and the testing datasets
df = scparadise.scnoah.cell_counter(adata, celltype='cell_type', sample='donor_id')
df

[6]:
HN481 HN482 HN483 HN485 HN487 HN488 HN489 HN490 HN492 HN494
epithelial cell 6747 758 934 575 2061 1335 2533 2955 2619 780
plasma cell 868 1051 498 609 106 596 1005 975 642 579
CD8-positive, alpha-beta T cell 613 1129 1204 740 279 941 1004 472 1408 911
fibroblast 512 164 256 163 371 118 856 204 234 560
myeloid cell 415 311 159 361 995 136 929 394 203 2998
regulatory T cell 370 316 623 351 135 69 881 385 321 616
endothelial cell 283 174 144 111 122 152 311 240 135 138
CD4-positive, alpha-beta T cell 282 1272 2629 911 87 224 1310 1076 789 1784
mural cell 267 170 98 156 195 173 286 233 87 128
B cell 213 1837 3369 764 30 2053 1338 2297 942 1832
mast cell 92 56 91 56 20 39 33 29 74 27
natural killer cell 88 135 205 65 40 120 465 94 58 128
mature alpha-beta T cell 44 31 13 7 8 7 82 30 38 42
endothelial cell of lymphatic vessel 29 23 25 10 8 9 59 6 28 8
plasmacytoid dendritic cell 19 31 43 25 13 22 87 24 20 163 
[7]:

# Create adata_train1, adata_train2 and adata_test datasets
adata_test = adata[adata.obs['donor_id'].isin(['HN481', 'HN482'])].copy()
adata_train_1 = adata[adata.obs['donor_id'].isin(['HN483' 'HN485', 'HN488', 'HN489'])].copy()
adata_train_2 = adata[adata.obs['donor_id'].isin(['HN490', 'HN487', 'HN492', 'HN494'])].copy()
del adata

[8]:

# Normalize data
for i in [adata_train_1, adata_train_2, adata_test]:
    i.layers['counts'] = i.X.copy()
    sc.pp.normalize_total(i, target_sum=None)
    sc.pp.log1p(i)

[9]:

# Find highly variable features for scAdam model training
sc.pp.highly_variable_genes(
    adata_train_1,
    layer='counts',
    flavor='seurat_v3',
    n_top_genes=1200,
    subset=True
)

Default model training#

Use ‘balanced_accuracy’ as the evaluation metric in this case due to the imbalance in the number of cells among cell types.

[10]:

# Train scadam model using adata_train1 dataset
scparadise.scadam.train(
    adata_train_1,
    path='', # path to save model
    model_name='model_scadam', # folder name with model
    celltype_keys=['cell_type'],
    eval_metric=['accuracy', 'balanced_accuracy']
)

Device: cuda
Number of features: 1200
Label hierarchy: cell_type
Annotation levels weights using strategy 'linear_offset':
  cell_type: 15 cell types, 1.0 relative weight

Dataset split:
Train dataset contains: 13738 cells, it is 80.0 % of input dataset
Validation dataset contains: 3435 cells, it is 20.0 % of input dataset

Unsupervised pretraining: 100%|████████████████| 50/50 [02:26<00:00,  2.92s/it]
Training scAdam model:  10%|█▉                | 21/200 [00:56<08:00,  2.68s/it]

Early stopping triggered! Best score: 0.9534
Training completed!
Fitting unknown cells detector



UnknownCellDetector fitted successfully!
Model saved to model_scadam

[11]:

# Predict cell types using trained model
adata_test = scparadise.scadam.predict(
    adata_test,
    path_model = 'model_scadam'
)

Cell type annotation using the 2nd version of the scAdam model
scAdam model with unknown detector loaded from model_scadam
Gene alignment:
  Model features: 1200
  Matched features: 1200 (100.0%)

Predicting: 100%|█████████████████████████████| 72/72 [00:00<00:00, 160.62it/s]

Added cell type column: pred_celltype_l1
Added probabilities column: pred_celltype_l1_probability

Evaluation of prediction quality#

The model cannot predict mature alpha-beta T cell (precision = 0.1199).

Also model cannot predict 50% of epithelial cell (recall/sensitivity = 0.5229).

There is also low sensitivity for regulatory T cell and low precision for CD8-positive and CD4-positive alpha-beta T cells, regulatory T cell, B cell.

[12]:

## Check model quality
df = scparadise.scnoah.report_classif_full(
    adata_test,
    celltype='cell_type',
    pred_celltype='pred_celltype_l1'
)
df

[12]:
precision recall/sensitivity specificity f1-score geometric mean index balanced accuracy number of cells
B cell 0.6606 0.982 0.9364 0.7899 0.9589 0.9237 2050
CD4-positive, alpha-beta T cell 0.4553 0.816 0.9094 0.5845 0.8614 0.7351 1554
CD8-positive, alpha-beta T cell 0.7178 0.876 0.9638 0.789 0.9188 0.8369 1742
endothelial cell 1.0000 0.9912 1.0 0.9956 0.9956 0.9904 457
endothelial cell of lymphatic vessel 1.0000 0.9615 1.0 0.9804 0.9806 0.9578 52
epithelial cell 0.9975 0.5229 0.9991 0.6861 0.7227 0.4975 7505
fibroblast 0.9864 0.9689 0.9995 0.9776 0.9841 0.9655 676
mast cell 0.9793 0.9595 0.9998 0.9693 0.9794 0.9554 148
mature alpha-beta T cell 0.1199 0.9067 0.9726 0.2118 0.9391 0.876 75
mural cell 0.9554 0.9794 0.9989 0.9672 0.9891 0.9764 437
myeloid cell 0.9781 0.9862 0.9991 0.9822 0.9926 0.9841 726
natural killer cell 0.5815 0.9596 0.9915 0.7242 0.9754 0.9484 223
plasma cell 0.9647 0.9979 0.9957 0.981 0.9968 0.9939 1919
plasmacytoid dendritic cell 0.9800 0.98 0.9999 0.98 0.9899 0.978 50
regulatory T cell 0.5116 0.7055 0.9738 0.5931 0.8289 0.6686 686
macro avg 0.7926 0.9062 0.9826 0.8141 0.9409 0.8858
weighted avg 0.8545 0.7598 0.9796 0.7657 0.8534 0.7299
Accuracy 0.7598
Balanced accuracy 0.9062 
[13]:

# Order cell type colors
celltype = np.unique(adata_test.obs['cell_type']).tolist()
adata_test.obs['cell_type'] = pd.Categorical(
    values=adata_test.obs['cell_type'], categories=celltype, ordered=True
)
adata_test.obs['pred_celltype_l1'] = pd.Categorical(
    values=adata_test.obs['pred_celltype_l1'], categories=celltype, ordered=True
)
scparadise.scnoah.pred_status(adata_test, celltype='cell_type', pred_celltype='pred_celltype_l1')

[14]:

# Visualise predicted cell types levels, prediction probabilities and prediction status
sc.pl.umap(
    adata_test,
    color=[
        'cell_type',
        'pred_celltype_l1',
        'pred_celltype_l1_probability',
        'pred_status'
    ],
    frameon = False,
    add_outline = True,
    legend_loc = 'right margin',
    legend_fontsize = 7,
    legend_fontoutline = 1,
    ncols=2,
    wspace = 0.7,
    hspace = 0.1
)
../../../_images/tutorials_notebooks_scAdam_scAdam_model_optimization_19_0.png

Warm start model training#

Warning! For warm start scAdam model training, a dataset is required that contains all the same cell types that were present in the original training. The cell types must be named exactly as they were during the initial training of the model.

  1. There should be no new additional cell types or levels of annotation.

  2. Additionally, none of the cell types used for the initial training of the model should be missing.

[15]:

# Warm start requires second training dataset and path to pretrained model
scparadise.scadam.warm_start(
    adata_train_2,
    path='', # path to save model
    path_model='model_scadam', # folder name with pretrained model
    model_name='model_scadam_warm_start', # folder name with fine-tuned model
    celltype_keys=['cell_type'],
    eval_metric=['accuracy', 'balanced_accuracy']
)

scAdam model with unknown detector loaded from model_scadam
Device: cuda
Gene alignment:
 Number of features: 1200
 Matched features: 1200 (100.0%)
Label hierarchy: cell_type
Annotation levels weights using strategy 'linear_offset':
  cell_type: 15 cell types, 1.0 relative weight

Dataset split:
Train dataset contains: 25740 cells, it is 80.0 % of input dataset
Validation dataset contains: 6436 cells, it is 20.0 % of input dataset

Warm-start model fine-tuning:  21%|██▎        | 21/100 [01:48<06:49,  5.18s/it]

Early stopping triggered! Best score: 0.9513
Fitting unknown cells detector



UnknownCellDetector fitted successfully!
Model saved to model_scadam_warm_start

[16]:

# Predict cell types using tuned model
adata_test = scparadise.scadam.predict(
    adata_test,
    path_model = 'model_scadam_warm_start'
)

Cell type annotation using the 2nd version of the scAdam model
scAdam model with unknown detector loaded from model_scadam_warm_start
Gene alignment:
  Model features: 1200
  Matched features: 1200 (100.0%)

Predicting: 100%|█████████████████████████████| 72/72 [00:00<00:00, 178.42it/s]

Added cell type column: pred_celltype_l1
Added probabilities column: pred_celltype_l1_probability

Evaluation of prediction results#

Warm start leads to an increase in the model’s accuracy (+17.1%), balanced accuracy (+4.3%), precision (+9.5%), f1-score (+10.2%), geometric mean (+3%), and index balanced accuracy (+5.5%).

Below are examples of improved model performance for certain cell types.

  1. mature alpha-beta T cell (precision - 0.6837 vs 0.1199)

  2. regulatory T cell (precision - 0.7475 vs 0.5116)

  3. epithelial cell (sensitivity - 0.9800 vs 0.5229)

  4. CD4-positive, alpha-beta T cell (precision - 0.8309 vs 0.4553)

  5. CD8-positive, alpha-beta T cell (precision - 0.9252 vs 0.7178)

Full comparison is available below:

[17]:

## Check model quality
df_warm_start = scparadise.scnoah.report_classif_full(
    adata_test,
    celltype='cell_type',
    pred_celltype='pred_celltype_l1'
)
df_warm_start

[17]:
precision recall/sensitivity specificity f1-score geometric mean index balanced accuracy number of cells
B cell 0.9906 0.9771 0.9988 0.9838 0.9879 0.9738 2050
CD4-positive, alpha-beta T cell 0.8309 0.8571 0.9838 0.8438 0.9183 0.8326 1554
CD8-positive, alpha-beta T cell 0.9252 0.9156 0.9922 0.9204 0.9531 0.9015 1742
endothelial cell 0.9956 0.9934 0.9999 0.9945 0.9967 0.9927 457
endothelial cell of lymphatic vessel 1.0000 1.0 1.0 1.0 1.0 1.0 52
epithelial cell 0.9930 0.98 0.9952 0.9865 0.9876 0.9738 7505
fibroblast 0.9938 0.9556 0.9998 0.9744 0.9774 0.9512 676
mast cell 0.9931 0.973 0.9999 0.9829 0.9864 0.9703 148
mature alpha-beta T cell 0.6837 0.8933 0.9983 0.7746 0.9444 0.8825 75
mural cell 0.9272 0.9908 0.9981 0.958 0.9945 0.9882 437
myeloid cell 0.9876 0.9848 0.9995 0.9862 0.9921 0.9829 726
natural killer cell 0.8220 0.9731 0.9974 0.8912 0.9852 0.9682 223
plasma cell 0.9912 0.9958 0.999 0.9935 0.9974 0.9945 1919
plasmacytoid dendritic cell 0.9608 0.98 0.9999 0.9703 0.9899 0.9779 50
regulatory T cell 0.7475 0.7726 0.9898 0.7599 0.8745 0.7481 686
macro avg 0.9228 0.9495 0.9968 0.9347 0.9724 0.9426
weighted avg 0.9580 0.9564 0.9952 0.957 0.9753 0.9486
Accuracy 0.9564
Balanced accuracy 0.9495 
[18]:

# Order cell type colors
celltype = np.unique(adata_test.obs['cell_type']).tolist()
adata_test.obs['cell_type'] = pd.Categorical(
    values=adata_test.obs['cell_type'], categories=celltype, ordered=True
)
adata_test.obs['pred_celltype_l1'] = pd.Categorical(
    values=adata_test.obs['pred_celltype_l1'], categories=celltype, ordered=True
)
# Add prediction status. Label cells as correct or incorrect based on the comparison between ground truth cell types and predictions.
scparadise.scnoah.pred_status(adata_test, celltype='cell_type', pred_celltype='pred_celltype_l1')

[19]:

# Visualise predicted cell types levels, prediction probabilities and prediction status
sc.pl.umap(
    adata_test,
    color=[
        'cell_type',
        'pred_celltype_l1',
        'pred_celltype_l1_probability',
        'pred_status'
    ],
    frameon = False,
    add_outline = True,
    legend_loc = 'right margin',
    legend_fontsize = 7,
    legend_fontoutline = 1,
    ncols=2,
    wspace = 0.7,
    hspace = 0.1
)
../../../_images/tutorials_notebooks_scAdam_scAdam_model_optimization_27_0.png 
[20]:

# Compare prediction results
# 'untuned' row represents untuned model
# 'warm start' row represents warm start trained model
df.compare(df_warm_start, keep_equal=True, align_axis = 0, result_names=('untuned', 'warm start'))

[20]:
precision recall/sensitivity specificity f1-score geometric mean index balanced accuracy
B cell untuned 0.6606 0.982 0.9364 0.7899 0.9589 0.9237
warm start 0.9906 0.9771 0.9988 0.9838 0.9879 0.9738
CD4-positive, alpha-beta T cell untuned 0.4553 0.816 0.9094 0.5845 0.8614 0.7351
warm start 0.8309 0.8571 0.9838 0.8438 0.9183 0.8326
CD8-positive, alpha-beta T cell untuned 0.7178 0.876 0.9638 0.789 0.9188 0.8369
warm start 0.9252 0.9156 0.9922 0.9204 0.9531 0.9015
endothelial cell untuned 1.0000 0.9912 1.0 0.9956 0.9956 0.9904
warm start 0.9956 0.9934 0.9999 0.9945 0.9967 0.9927
endothelial cell of lymphatic vessel untuned 1.0000 0.9615 1.0 0.9804 0.9806 0.9578
warm start 1.0000 1.0 1.0 1.0 1.0 1.0
epithelial cell untuned 0.9975 0.5229 0.9991 0.6861 0.7227 0.4975
warm start 0.9930 0.98 0.9952 0.9865 0.9876 0.9738
fibroblast untuned 0.9864 0.9689 0.9995 0.9776 0.9841 0.9655
warm start 0.9938 0.9556 0.9998 0.9744 0.9774 0.9512
mast cell untuned 0.9793 0.9595 0.9998 0.9693 0.9794 0.9554
warm start 0.9931 0.973 0.9999 0.9829 0.9864 0.9703
mature alpha-beta T cell untuned 0.1199 0.9067 0.9726 0.2118 0.9391 0.876
warm start 0.6837 0.8933 0.9983 0.7746 0.9444 0.8825
mural cell untuned 0.9554 0.9794 0.9989 0.9672 0.9891 0.9764
warm start 0.9272 0.9908 0.9981 0.958 0.9945 0.9882
myeloid cell untuned 0.9781 0.9862 0.9991 0.9822 0.9926 0.9841
warm start 0.9876 0.9848 0.9995 0.9862 0.9921 0.9829
natural killer cell untuned 0.5815 0.9596 0.9915 0.7242 0.9754 0.9484
warm start 0.8220 0.9731 0.9974 0.8912 0.9852 0.9682
plasma cell untuned 0.9647 0.9979 0.9957 0.981 0.9968 0.9939
warm start 0.9912 0.9958 0.999 0.9935 0.9974 0.9945
plasmacytoid dendritic cell untuned 0.9800 0.98 0.9999 0.98 0.9899 0.978
warm start 0.9608 0.98 0.9999 0.9703 0.9899 0.9779
regulatory T cell untuned 0.5116 0.7055 0.9738 0.5931 0.8289 0.6686
warm start 0.7475 0.7726 0.9898 0.7599 0.8745 0.7481
macro avg untuned 0.7926 0.9062 0.9826 0.8141 0.9409 0.8858
warm start 0.9228 0.9495 0.9968 0.9347 0.9724 0.9426
weighted avg untuned 0.8545 0.7598 0.9796 0.7657 0.8534 0.7299
warm start 0.9580 0.9564 0.9952 0.957 0.9753 0.9486
Accuracy untuned 0.7598
warm start 0.9564
Balanced accuracy untuned 0.9062
warm start 0.9495

Hyperparameters tuning#

Warning! Hyperparameter tuning is a very time-consuming process and may be interrupted for various reasons (such as system shutdowns or errors in CUDA). However, this is not a problem, as restarting the hyperparameter tuning will allow it to continue from the last unfinished trial. An example of hyperparameter tuning represents such a case. The fourth line states: “Using an existing study with name ‘model_scadam_hp_tune’ instead of creating a new one.” This indicates a restart of hyperparameter tuning (in this case, from the 8th trial).

[21]:

scparadise.scadam.hyperparameter_tuning(
    adata_train_1,
    path='',
    model_name='model_scadam_hp_tune', # Folder to save hyperparameter tuning results
    celltype_keys=['cell_type'],
    num_trials=100, # The number of attempts to find the optimal hyperparameters for the model (recommended - minimum 100)
    eval_metric=['accuracy', 'balanced_accuracy']
)

Device: cuda
Number of features: 1200
Number of cells: 17173
Label hierarchy: cell_type
Annotation levels weights using strategy 'linear_offset':
  cell_type: 15 cell types, 1.0 relative weight
Start model optimization using optuna...

[I 2026-09-03 22:29:55,814] A new study created in RDB with name: study

Fold 1 finished with balanced_accuracy value = 0.950665
Fold 2 finished with balanced_accuracy value = 0.954470
Fold 3 finished with balanced_accuracy value = 0.958962
Fold 4 finished with balanced_accuracy value = 0.962636
Fold 5 finished with balanced_accuracy value = 0.962258

[I 2026-09-03 22:48:18,344] Trial 0 finished with value: 0.9577981523587649 and parameters: {'nc': 16, 'nb': 5, 'nh': 8, 'ed_nh_ratio': 32, 'ff_hd': 512, 'classifier_hd': 256, 'dropout': 0.3, 'lr': 0.0001, 'weight_decay': 0.0001, 'batch_size': 128, 'patience': 10, 'epochs': 200, 'use_augmentation': True, 'aug_probability': 0.5, 'prob': 0.15, 'noise_std': 0.1, 'dropout_aug': 0.1, 'alpha': 0.2, 'from_unsupervised': True, 'pretrain_epochs': 50}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.559282
Fold 2 finished with balanced_accuracy value = 0.066667
Fold 3 finished with balanced_accuracy value = 0.778884
Fold 4 finished with balanced_accuracy value = 0.865783
Fold 5 finished with balanced_accuracy value = 0.737923

[I 2026-09-03 22:50:56,092] Trial 1 finished with value: 0.6017076628401339 and parameters: {'nc': 12, 'nb': 6, 'nh': 14, 'ed_nh_ratio': 20, 'ff_hd': 256, 'classifier_hd': 512, 'dropout': 0.0783889458904844, 'lr': 0.009508308324249943, 'weight_decay': 1.9355755643719093e-06, 'batch_size': 1920, 'patience': 5, 'epochs': 80, 'use_augmentation': False, 'aug_probability': 0.9643136361986554, 'prob': 0.3834679035334626, 'noise_std': 0.31507212684040786, 'dropout_aug': 0.2867972175182382, 'alpha': 0.16602679707525195, 'from_unsupervised': False, 'pretrain_epochs': 60}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.947738
Fold 2 finished with balanced_accuracy value = 0.951062
Fold 3 finished with balanced_accuracy value = 0.952015
Fold 4 finished with balanced_accuracy value = 0.956677
Fold 5 finished with balanced_accuracy value = 0.958757

[I 2026-09-03 22:57:05,094] Trial 2 finished with value: 0.9532500242886087 and parameters: {'nc': 6, 'nb': 8, 'nh': 4, 'ed_nh_ratio': 24, 'ff_hd': 384, 'classifier_hd': 896, 'dropout': 0.19087986708563537, 'lr': 0.00039747627056637954, 'weight_decay': 8.59480186158519e-06, 'batch_size': 1216, 'patience': 25, 'epochs': 55, 'use_augmentation': False, 'aug_probability': 0.4406228682533382, 'prob': 0.32449588113959016, 'noise_std': 0.28761133764513963, 'dropout_aug': 0.23525357782685555, 'alpha': 0.04807427325580163, 'from_unsupervised': True, 'pretrain_epochs': 75}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.954514
Fold 2 finished with balanced_accuracy value = 0.957209
Fold 3 finished with balanced_accuracy value = 0.955665
Fold 4 finished with balanced_accuracy value = 0.952211
Fold 5 finished with balanced_accuracy value = 0.957096

[I 2026-09-03 22:58:47,491] Trial 3 finished with value: 0.955338958425689 and parameters: {'nc': 8, 'nb': 1, 'nh': 16, 'ed_nh_ratio': 20, 'ff_hd': 512, 'classifier_hd': 640, 'dropout': 0.4390383775298243, 'lr': 0.00039362082431589784, 'weight_decay': 1.1589344512546852e-06, 'batch_size': 640, 'patience': 20, 'epochs': 90, 'use_augmentation': False, 'aug_probability': 0.3947816815559372, 'prob': 0.173369887772515, 'noise_std': 0.32059953806872, 'dropout_aug': 0.3136812318415079, 'alpha': 0.30988997020959624, 'from_unsupervised': False, 'pretrain_epochs': 10}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.574739
Fold 2 finished with balanced_accuracy value = 0.638022
Fold 3 finished with balanced_accuracy value = 0.565991

[I 2026-09-03 23:01:00,523] Trial 4 pruned.

Fold 1 finished with balanced_accuracy value = 0.947342
Fold 2 finished with balanced_accuracy value = 0.943976
Fold 3 finished with balanced_accuracy value = 0.945844
Fold 4 finished with balanced_accuracy value = 0.945477
Fold 5 finished with balanced_accuracy value = 0.952547

[I 2026-09-03 23:12:05,079] Trial 5 finished with value: 0.947037360382415 and parameters: {'nc': 8, 'nb': 3, 'nh': 10, 'ed_nh_ratio': 32, 'ff_hd': 512, 'classifier_hd': 256, 'dropout': 0.19907589684339705, 'lr': 1.39038098658704e-05, 'weight_decay': 0.007381756757706973, 'batch_size': 384, 'patience': 25, 'epochs': 135, 'use_augmentation': True, 'aug_probability': 0.9622345817018797, 'prob': 0.3174181629814361, 'noise_std': 0.19046224881808316, 'dropout_aug': 0.11449668115546148, 'alpha': 0.16878964934243923, 'from_unsupervised': True, 'pretrain_epochs': 50}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.936148

[I 2026-09-03 23:18:00,528] Trial 6 pruned.

Fold 1 finished with balanced_accuracy value = 0.951958
Fold 2 finished with balanced_accuracy value = 0.950918
Fold 3 finished with balanced_accuracy value = 0.961305
Fold 4 finished with balanced_accuracy value = 0.953708
Fold 5 finished with balanced_accuracy value = 0.964094

[I 2026-09-03 23:21:02,252] Trial 7 finished with value: 0.9563966082933513 and parameters: {'nc': 2, 'nb': 7, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 1024, 'classifier_hd': 896, 'dropout': 0.3447249772238844, 'lr': 0.00010103921964115245, 'weight_decay': 0.004175030920352183, 'batch_size': 1472, 'patience': 25, 'epochs': 155, 'use_augmentation': True, 'aug_probability': 0.7389200291345882, 'prob': 0.19586773310490935, 'noise_std': 0.3579592762647379, 'dropout_aug': 0.2745532376006321, 'alpha': 0.006627637430921674, 'from_unsupervised': False, 'pretrain_epochs': 40}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.922172
Fold 2 finished with balanced_accuracy value = 0.945469
Fold 3 finished with balanced_accuracy value = 0.943373

[I 2026-09-03 23:25:27,095] Trial 8 pruned.

Fold 1 finished with balanced_accuracy value = 0.941606

[I 2026-09-03 23:26:20,377] Trial 9 pruned.

Fold 1 finished with balanced_accuracy value = 0.850490
Fold 2 finished with balanced_accuracy value = 0.844920
Fold 3 finished with balanced_accuracy value = 0.850382

[I 2026-09-03 23:29:30,727] Trial 10 pruned.

Fold 1 finished with balanced_accuracy value = 0.951824
Fold 2 finished with balanced_accuracy value = 0.956568
Fold 3 finished with balanced_accuracy value = 0.957922

[I 2026-09-03 23:30:53,395] Trial 11 pruned.

Fold 1 finished with balanced_accuracy value = 0.955451
Fold 2 finished with balanced_accuracy value = 0.952330
Fold 3 finished with balanced_accuracy value = 0.957164
Fold 4 finished with balanced_accuracy value = 0.960872
Fold 5 finished with balanced_accuracy value = 0.961601

[I 2026-09-03 23:34:06,602] Trial 12 finished with value: 0.9574834597638386 and parameters: {'nc': 2, 'nb': 5, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 768, 'classifier_hd': 768, 'dropout': 0.3334314498964346, 'lr': 4.480139375110715e-05, 'weight_decay': 0.0011628507511024413, 'batch_size': 832, 'patience': 30, 'epochs': 175, 'use_augmentation': True, 'aug_probability': 0.6248951090819718, 'prob': 0.24248650983506762, 'noise_std': 0.09942303272518321, 'dropout_aug': 0.12813713951088232, 'alpha': 0.08646878144364928, 'from_unsupervised': False, 'pretrain_epochs': 40}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.951288
Fold 2 finished with balanced_accuracy value = 0.953362
Fold 3 finished with balanced_accuracy value = 0.959997
Fold 4 finished with balanced_accuracy value = 0.952626
Fold 5 finished with balanced_accuracy value = 0.953743

[I 2026-09-03 23:41:36,769] Trial 13 finished with value: 0.954203066494213 and parameters: {'nc': 12, 'nb': 5, 'nh': 12, 'ed_nh_ratio': 28, 'ff_hd': 768, 'classifier_hd': 640, 'dropout': 0.29199569843517253, 'lr': 3.7591482786882805e-05, 'weight_decay': 0.0008114987875168701, 'batch_size': 832, 'patience': 30, 'epochs': 170, 'use_augmentation': True, 'aug_probability': 0.5316773258800174, 'prob': 0.26092321806338686, 'noise_std': 0.08392815072551195, 'dropout_aug': 0.10968909176202524, 'alpha': 0.11132854299502683, 'from_unsupervised': False, 'pretrain_epochs': 25}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.948517

[I 2026-09-03 23:48:20,147] Trial 14 pruned.

Fold 1 finished with balanced_accuracy value = 0.946513
Fold 2 finished with balanced_accuracy value = 0.955095
Fold 3 finished with balanced_accuracy value = 0.959059
Fold 4 finished with balanced_accuracy value = 0.950363
Fold 5 finished with balanced_accuracy value = 0.954707

[I 2026-09-03 23:54:19,541] Trial 15 finished with value: 0.9531473451054607 and parameters: {'nc': 12, 'nb': 5, 'nh': 6, 'ed_nh_ratio': 28, 'ff_hd': 384, 'classifier_hd': 768, 'dropout': 0.26312159803870927, 'lr': 3.347331284748137e-05, 'weight_decay': 0.0008102085950014221, 'batch_size': 576, 'patience': 15, 'epochs': 180, 'use_augmentation': True, 'aug_probability': 0.3359906726105064, 'prob': 0.2526323464979391, 'noise_std': 0.1214848052698469, 'dropout_aug': 0.0601816634401601, 'alpha': 0.09441232566159813, 'from_unsupervised': False, 'pretrain_epochs': 45}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.956605
Fold 2 finished with balanced_accuracy value = 0.951144
Fold 3 finished with balanced_accuracy value = 0.956837

[I 2026-09-03 23:55:16,870] Trial 16 pruned.

Fold 1 finished with balanced_accuracy value = 0.947744

[I 2026-09-03 23:56:03,982] Trial 17 pruned.

Fold 1 finished with balanced_accuracy value = 0.949558
Fold 2 finished with balanced_accuracy value = 0.956925
Fold 3 finished with balanced_accuracy value = 0.956239

[I 2026-09-03 23:58:31,229] Trial 18 pruned.

Fold 1 finished with balanced_accuracy value = 0.946270

[I 2026-09-04 00:00:04,749] Trial 19 pruned.

Fold 1 finished with balanced_accuracy value = 0.578455

[I 2026-09-04 00:01:22,727] Trial 20 pruned.

Fold 1 finished with balanced_accuracy value = 0.951118
Fold 2 finished with balanced_accuracy value = 0.953670
Fold 3 finished with balanced_accuracy value = 0.958296
Fold 4 finished with balanced_accuracy value = 0.954927
Fold 5 finished with balanced_accuracy value = 0.961199

[I 2026-09-04 00:04:43,070] Trial 21 finished with value: 0.9558417419204204 and parameters: {'nc': 2, 'nb': 8, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 896, 'classifier_hd': 896, 'dropout': 0.33877875439413124, 'lr': 8.892116474459765e-05, 'weight_decay': 0.003419925908371807, 'batch_size': 1664, 'patience': 25, 'epochs': 160, 'use_augmentation': True, 'aug_probability': 0.7036719709103834, 'prob': 0.18450441063095324, 'noise_std': 0.37513069138091504, 'dropout_aug': 0.39999707929961237, 'alpha': 0.021001763037931977, 'from_unsupervised': False, 'pretrain_epochs': 40}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.954239
Fold 2 finished with balanced_accuracy value = 0.951026
Fold 3 finished with balanced_accuracy value = 0.955118

[I 2026-09-04 00:05:54,156] Trial 22 pruned.

Fold 1 finished with balanced_accuracy value = 0.951311
Fold 2 finished with balanced_accuracy value = 0.952599
Fold 3 finished with balanced_accuracy value = 0.961189
Fold 4 finished with balanced_accuracy value = 0.950650
Fold 5 finished with balanced_accuracy value = 0.962696

[I 2026-09-04 00:12:12,273] Trial 23 finished with value: 0.9556888519224932 and parameters: {'nc': 6, 'nb': 6, 'nh': 14, 'ed_nh_ratio': 32, 'ff_hd': 896, 'classifier_hd': 1024, 'dropout': 0.321314312351882, 'lr': 2.2381170077745966e-05, 'weight_decay': 0.0008333033601423902, 'batch_size': 1024, 'patience': 25, 'epochs': 170, 'use_augmentation': True, 'aug_probability': 0.6174367936087574, 'prob': 0.14512386093521884, 'noise_std': 0.03535816521324965, 'dropout_aug': 0.24260942790822376, 'alpha': 0.12605215531882566, 'from_unsupervised': False, 'pretrain_epochs': 50}. Best is trial 0 with value: 0.9577981523587649.

Fold 1 finished with balanced_accuracy value = 0.954320
Fold 2 finished with balanced_accuracy value = 0.962420
Fold 3 finished with balanced_accuracy value = 0.959860
Fold 4 finished with balanced_accuracy value = 0.961410
Fold 5 finished with balanced_accuracy value = 0.969924

[I 2026-09-04 00:16:07,542] Trial 24 finished with value: 0.9615867721230634 and parameters: {'nc': 4, 'nb': 5, 'nh': 16, 'ed_nh_ratio': 28, 'ff_hd': 768, 'classifier_hd': 896, 'dropout': 0.23698142213184742, 'lr': 6.963690736722135e-05, 'weight_decay': 0.0034732362400466494, 'batch_size': 1408, 'patience': 30, 'epochs': 195, 'use_augmentation': True, 'aug_probability': 0.7677578787467152, 'prob': 0.20685827234760135, 'noise_std': 0.10838449541841462, 'dropout_aug': 0.00177856974742846, 'alpha': 0.04645915647446934, 'from_unsupervised': False, 'pretrain_epochs': 25}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.954038
Fold 2 finished with balanced_accuracy value = 0.957207
Fold 3 finished with balanced_accuracy value = 0.958559
Fold 4 finished with balanced_accuracy value = 0.954125
Fold 5 finished with balanced_accuracy value = 0.957720

[I 2026-09-04 00:24:07,049] Trial 25 finished with value: 0.9563298508533247 and parameters: {'nc': 4, 'nb': 5, 'nh': 10, 'ed_nh_ratio': 24, 'ff_hd': 640, 'classifier_hd': 512, 'dropout': 0.24494925927751768, 'lr': 5.717831519044033e-05, 'weight_decay': 0.0003551534779952667, 'batch_size': 192, 'patience': 30, 'epochs': 190, 'use_augmentation': True, 'aug_probability': 0.8718077691461643, 'prob': 0.08469132480883367, 'noise_std': 0.12267509943127838, 'dropout_aug': 0.0018726663061699694, 'alpha': 0.04267917575401342, 'from_unsupervised': False, 'pretrain_epochs': 20}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.947334

[I 2026-09-04 00:26:08,376] Trial 26 pruned.

Fold 1 finished with balanced_accuracy value = 0.949740
Fold 2 finished with balanced_accuracy value = 0.952683
Fold 3 finished with balanced_accuracy value = 0.956463

[I 2026-09-04 00:28:43,989] Trial 27 pruned.

Fold 1 finished with balanced_accuracy value = 0.955329
Fold 2 finished with balanced_accuracy value = 0.953746
Fold 3 finished with balanced_accuracy value = 0.956160

[I 2026-09-04 00:30:44,728] Trial 28 pruned.

Fold 1 finished with balanced_accuracy value = 0.903565

[I 2026-09-04 00:31:11,280] Trial 29 pruned.

Fold 1 finished with balanced_accuracy value = 0.950290

[I 2026-09-04 00:31:26,920] Trial 30 pruned.

Fold 1 finished with balanced_accuracy value = 0.954240
Fold 2 finished with balanced_accuracy value = 0.952897
Fold 3 finished with balanced_accuracy value = 0.959349
Fold 4 finished with balanced_accuracy value = 0.951945
Fold 5 finished with balanced_accuracy value = 0.960417

[I 2026-09-04 00:33:49,535] Trial 31 finished with value: 0.9557695794700101 and parameters: {'nc': 2, 'nb': 7, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 1024, 'classifier_hd': 896, 'dropout': 0.35815979203289244, 'lr': 0.00012382864687733762, 'weight_decay': 0.0056557939298423725, 'batch_size': 1408, 'patience': 30, 'epochs': 150, 'use_augmentation': True, 'aug_probability': 0.7336887233087528, 'prob': 0.19622708318373952, 'noise_std': 0.09847418662442144, 'dropout_aug': 0.24689926923091318, 'alpha': 0.03705881789180691, 'from_unsupervised': False, 'pretrain_epochs': 40}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.953955
Fold 2 finished with balanced_accuracy value = 0.951800
Fold 3 finished with balanced_accuracy value = 0.959264

[I 2026-09-04 00:35:15,396] Trial 32 pruned.

Fold 1 finished with balanced_accuracy value = 0.955150
Fold 2 finished with balanced_accuracy value = 0.952947
Fold 3 finished with balanced_accuracy value = 0.957031

[I 2026-09-04 00:37:32,896] Trial 33 pruned.

Fold 1 finished with balanced_accuracy value = 0.951059
Fold 2 finished with balanced_accuracy value = 0.955822
Fold 3 finished with balanced_accuracy value = 0.957640

[I 2026-09-04 00:39:02,295] Trial 34 pruned.

Fold 1 finished with balanced_accuracy value = 0.947895
Fold 2 finished with balanced_accuracy value = 0.953615
Fold 3 finished with balanced_accuracy value = 0.959315
Fold 4 finished with balanced_accuracy value = 0.950717
Fold 5 finished with balanced_accuracy value = 0.963923

[I 2026-09-04 00:40:45,889] Trial 35 finished with value: 0.9550931209018245 and parameters: {'nc': 8, 'nb': 2, 'nh': 14, 'ed_nh_ratio': 32, 'ff_hd': 384, 'classifier_hd': 896, 'dropout': 0.42766852878494294, 'lr': 0.0002765624421711542, 'weight_decay': 0.0012053414474107302, 'batch_size': 1152, 'patience': 25, 'epochs': 125, 'use_augmentation': True, 'aug_probability': 0.7855878659565504, 'prob': 0.18745626802116236, 'noise_std': 0.051678238837171014, 'dropout_aug': 0.17508812717639405, 'alpha': 0.0230780499629716, 'from_unsupervised': False, 'pretrain_epochs': 60}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.950769

[I 2026-09-04 00:42:20,640] Trial 36 pruned.

Fold 1 finished with balanced_accuracy value = 0.918171

[I 2026-09-04 00:42:51,229] Trial 37 pruned.

Fold 1 finished with balanced_accuracy value = 0.949116

[I 2026-09-04 00:44:54,897] Trial 38 pruned.

Fold 1 finished with balanced_accuracy value = 0.950501

[I 2026-09-04 00:46:34,493] Trial 39 pruned.

Fold 1 finished with balanced_accuracy value = 0.951727

[I 2026-09-04 00:48:01,759] Trial 40 pruned.

Fold 1 finished with balanced_accuracy value = 0.952236
Fold 2 finished with balanced_accuracy value = 0.958239
Fold 3 finished with balanced_accuracy value = 0.957486

[I 2026-09-04 00:56:33,675] Trial 41 pruned.

Fold 1 finished with balanced_accuracy value = 0.950311

[I 2026-09-04 00:58:41,966] Trial 42 pruned.

Fold 1 finished with balanced_accuracy value = 0.956383
Fold 2 finished with balanced_accuracy value = 0.952114
Fold 3 finished with balanced_accuracy value = 0.956385

[I 2026-09-04 01:02:21,852] Trial 43 pruned.

Fold 1 finished with balanced_accuracy value = 0.947590
Fold 2 finished with balanced_accuracy value = 0.951522
Fold 3 finished with balanced_accuracy value = 0.955331

[I 2026-09-04 01:14:02,858] Trial 44 pruned.

Fold 1 finished with balanced_accuracy value = 0.952013
Fold 2 finished with balanced_accuracy value = 0.953796
Fold 3 finished with balanced_accuracy value = 0.953830

[I 2026-09-04 01:19:47,031] Trial 45 pruned.

Fold 1 finished with balanced_accuracy value = 0.948512

[I 2026-09-04 01:20:18,499] Trial 46 pruned.

Fold 1 finished with balanced_accuracy value = 0.950480

[I 2026-09-04 01:20:48,619] Trial 47 pruned.

Fold 1 finished with balanced_accuracy value = 0.946757

[I 2026-09-04 01:21:54,266] Trial 48 pruned.

Fold 1 finished with balanced_accuracy value = 0.820752
Fold 2 finished with balanced_accuracy value = 0.761568
Fold 3 finished with balanced_accuracy value = 0.803931

[I 2026-09-04 01:25:58,700] Trial 49 pruned.

Fold 1 finished with balanced_accuracy value = 0.950648

[I 2026-09-04 01:26:13,868] Trial 50 pruned.

Fold 1 finished with balanced_accuracy value = 0.950946

[I 2026-09-04 01:26:49,790] Trial 51 pruned.

Fold 1 finished with balanced_accuracy value = 0.955543
Fold 2 finished with balanced_accuracy value = 0.955902
Fold 3 finished with balanced_accuracy value = 0.959155
Fold 4 finished with balanced_accuracy value = 0.956369
Fold 5 finished with balanced_accuracy value = 0.963711

[I 2026-09-04 01:30:02,829] Trial 52 finished with value: 0.9581359221110144 and parameters: {'nc': 2, 'nb': 8, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 768, 'classifier_hd': 896, 'dropout': 0.35225506394349737, 'lr': 5.15098166542104e-05, 'weight_decay': 0.007824861126045444, 'batch_size': 1536, 'patience': 25, 'epochs': 140, 'use_augmentation': True, 'aug_probability': 0.6869718620455656, 'prob': 0.1778135263078322, 'noise_std': 0.36439654870008825, 'dropout_aug': 0.369024110531363, 'alpha': 0.03379174806361271, 'from_unsupervised': False, 'pretrain_epochs': 45}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.951084
Fold 2 finished with balanced_accuracy value = 0.955827
Fold 3 finished with balanced_accuracy value = 0.957841
Fold 4 finished with balanced_accuracy value = 0.958228
Fold 5 finished with balanced_accuracy value = 0.957559

[I 2026-09-04 01:33:10,643] Trial 53 finished with value: 0.956107714537947 and parameters: {'nc': 4, 'nb': 7, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 640, 'classifier_hd': 896, 'dropout': 0.367167166437394, 'lr': 5.211911874241945e-05, 'weight_decay': 0.007086454132669151, 'batch_size': 1536, 'patience': 20, 'epochs': 145, 'use_augmentation': True, 'aug_probability': 0.695477400909415, 'prob': 0.21820876493731642, 'noise_std': 0.3590997544085141, 'dropout_aug': 0.36206219644493165, 'alpha': 0.03399555956428839, 'from_unsupervised': False, 'pretrain_epochs': 50}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.953613
Fold 2 finished with balanced_accuracy value = 0.957094
Fold 3 finished with balanced_accuracy value = 0.960063
Fold 4 finished with balanced_accuracy value = 0.957473
Fold 5 finished with balanced_accuracy value = 0.964199

[I 2026-09-04 01:36:52,371] Trial 54 finished with value: 0.9584884237729888 and parameters: {'nc': 2, 'nb': 8, 'nh': 14, 'ed_nh_ratio': 32, 'ff_hd': 768, 'classifier_hd': 768, 'dropout': 0.28816486840673106, 'lr': 3.9485589849683886e-05, 'weight_decay': 0.0020155969088541535, 'batch_size': 896, 'patience': 30, 'epochs': 140, 'use_augmentation': True, 'aug_probability': 0.5222365238057864, 'prob': 0.17844126048625, 'noise_std': 0.3981552461981962, 'dropout_aug': 0.31175286634160504, 'alpha': 0.07432493130788397, 'from_unsupervised': False, 'pretrain_epochs': 45}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.954065
Fold 2 finished with balanced_accuracy value = 0.955554
Fold 3 finished with balanced_accuracy value = 0.959312

[I 2026-09-04 01:38:36,478] Trial 55 pruned.

Fold 1 finished with balanced_accuracy value = 0.955139
Fold 2 finished with balanced_accuracy value = 0.949846
Fold 3 finished with balanced_accuracy value = 0.959321
Fold 4 finished with balanced_accuracy value = 0.957798
Fold 5 finished with balanced_accuracy value = 0.963926

[I 2026-09-04 01:43:50,617] Trial 56 finished with value: 0.9572061317478091 and parameters: {'nc': 2, 'nb': 8, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 768, 'classifier_hd': 896, 'dropout': 0.40838328854555583, 'lr': 7.375529722524565e-05, 'weight_decay': 0.0021152841388750493, 'batch_size': 1152, 'patience': 30, 'epochs': 125, 'use_augmentation': True, 'aug_probability': 0.4718733816275893, 'prob': 0.20198335615781668, 'noise_std': 0.3907148989268455, 'dropout_aug': 0.3749530393269231, 'alpha': 0.011782139054995061, 'from_unsupervised': True, 'pretrain_epochs': 45}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.940035

[I 2026-09-04 01:47:34,266] Trial 57 pruned.

Fold 1 finished with balanced_accuracy value = 0.953848
Fold 2 finished with balanced_accuracy value = 0.951339
Fold 3 finished with balanced_accuracy value = 0.955264

[I 2026-09-04 01:50:35,989] Trial 58 pruned.

Fold 1 finished with balanced_accuracy value = 0.954262
Fold 2 finished with balanced_accuracy value = 0.955052
Fold 3 finished with balanced_accuracy value = 0.958194

[I 2026-09-04 01:55:08,608] Trial 59 pruned.

Fold 1 finished with balanced_accuracy value = 0.932338

[I 2026-09-04 01:56:15,943] Trial 60 pruned.

Fold 1 finished with balanced_accuracy value = 0.955963
Fold 2 finished with balanced_accuracy value = 0.950981
Fold 3 finished with balanced_accuracy value = 0.958029
Fold 4 finished with balanced_accuracy value = 0.958838
Fold 5 finished with balanced_accuracy value = 0.964213

[I 2026-09-04 02:01:00,711] Trial 61 finished with value: 0.95760479282514 and parameters: {'nc': 2, 'nb': 7, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 896, 'classifier_hd': 896, 'dropout': 0.3481479118184314, 'lr': 4.4769865793186965e-05, 'weight_decay': 0.0017336371017094628, 'batch_size': 1408, 'patience': 25, 'epochs': 140, 'use_augmentation': True, 'aug_probability': 0.5884152919197089, 'prob': 0.19352895679140564, 'noise_std': 0.3595830606261121, 'dropout_aug': 0.34158090632875887, 'alpha': 0.0112714631132305, 'from_unsupervised': True, 'pretrain_epochs': 40}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.947642
Fold 2 finished with balanced_accuracy value = 0.955031
Fold 3 finished with balanced_accuracy value = 0.959258
Fold 4 finished with balanced_accuracy value = 0.955291
Fold 5 finished with balanced_accuracy value = 0.961723

[I 2026-09-04 02:06:04,624] Trial 62 finished with value: 0.9557889532465144 and parameters: {'nc': 2, 'nb': 7, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 896, 'classifier_hd': 896, 'dropout': 0.3215263631855559, 'lr': 3.827442658583615e-05, 'weight_decay': 1.055407920567048e-06, 'batch_size': 1408, 'patience': 25, 'epochs': 140, 'use_augmentation': True, 'aug_probability': 0.4711263079997335, 'prob': 0.19285854984867215, 'noise_std': 0.36535663030504456, 'dropout_aug': 0.34866022657532303, 'alpha': 0.05308653740659563, 'from_unsupervised': True, 'pretrain_epochs': 40}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.950442
Fold 2 finished with balanced_accuracy value = 0.953210
Fold 3 finished with balanced_accuracy value = 0.959539
Fold 4 finished with balanced_accuracy value = 0.955607
Fold 5 finished with balanced_accuracy value = 0.965373

[I 2026-09-04 02:11:53,982] Trial 63 finished with value: 0.956834139577259 and parameters: {'nc': 2, 'nb': 8, 'nh': 14, 'ed_nh_ratio': 32, 'ff_hd': 768, 'classifier_hd': 768, 'dropout': 0.45634473324206726, 'lr': 4.6477718662334045e-05, 'weight_decay': 0.0018266663428673966, 'batch_size': 1152, 'patience': 30, 'epochs': 130, 'use_augmentation': True, 'aug_probability': 0.5895539661517901, 'prob': 0.2184502328095334, 'noise_std': 0.3961407645714306, 'dropout_aug': 0.3876774754913531, 'alpha': 0.014655060202556122, 'from_unsupervised': True, 'pretrain_epochs': 35}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.955139
Fold 2 finished with balanced_accuracy value = 0.951707
Fold 3 finished with balanced_accuracy value = 0.959696
Fold 4 finished with balanced_accuracy value = 0.960944
Fold 5 finished with balanced_accuracy value = 0.958362

[I 2026-09-04 02:17:05,792] Trial 64 finished with value: 0.9571695495494129 and parameters: {'nc': 2, 'nb': 7, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 768, 'classifier_hd': 1024, 'dropout': 0.3529644375417896, 'lr': 7.607657701048455e-05, 'weight_decay': 0.0006907854258664468, 'batch_size': 1344, 'patience': 25, 'epochs': 140, 'use_augmentation': True, 'aug_probability': 0.29502192729283727, 'prob': 0.20214221563355914, 'noise_std': 0.3511441514642375, 'dropout_aug': 0.37000383181264734, 'alpha': 0.02896956353844202, 'from_unsupervised': True, 'pretrain_epochs': 50}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.951442
Fold 2 finished with balanced_accuracy value = 0.952364
Fold 3 finished with balanced_accuracy value = 0.962681
Fold 4 finished with balanced_accuracy value = 0.952589
Fold 5 finished with balanced_accuracy value = 0.957726

[I 2026-09-04 02:26:08,600] Trial 65 finished with value: 0.9553603628320866 and parameters: {'nc': 4, 'nb': 8, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 896, 'classifier_hd': 896, 'dropout': 0.2784017570056487, 'lr': 2.0145452662671934e-05, 'weight_decay': 0.0016083337700043177, 'batch_size': 1472, 'patience': 20, 'epochs': 150, 'use_augmentation': True, 'aug_probability': 0.6275277759737825, 'prob': 0.17536167369045555, 'noise_std': 0.33005365742424947, 'dropout_aug': 0.34127416451408515, 'alpha': 0.07234842292068705, 'from_unsupervised': True, 'pretrain_epochs': 45}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.954390
Fold 2 finished with balanced_accuracy value = 0.955653
Fold 3 finished with balanced_accuracy value = 0.960147
Fold 4 finished with balanced_accuracy value = 0.956533
Fold 5 finished with balanced_accuracy value = 0.961210

[I 2026-09-04 02:29:35,658] Trial 66 finished with value: 0.9575863049975192 and parameters: {'nc': 2, 'nb': 6, 'nh': 14, 'ed_nh_ratio': 28, 'ff_hd': 768, 'classifier_hd': 768, 'dropout': 0.32161953289327244, 'lr': 0.00011327648606733358, 'weight_decay': 0.0033266569114842044, 'batch_size': 1216, 'patience': 30, 'epochs': 90, 'use_augmentation': True, 'aug_probability': 0.549624173052194, 'prob': 0.1545942230648168, 'noise_std': 0.3817356726830298, 'dropout_aug': 0.29911630501149156, 'alpha': 0.39401302894350476, 'from_unsupervised': True, 'pretrain_epochs': 35}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.951429

[I 2026-09-04 02:30:16,357] Trial 67 pruned.

Fold 1 finished with balanced_accuracy value = 0.951339

[I 2026-09-04 02:30:52,683] Trial 68 pruned.

Fold 1 finished with balanced_accuracy value = 0.925566

[I 2026-09-04 02:31:37,712] Trial 69 pruned.

Fold 1 finished with balanced_accuracy value = 0.950703
Fold 2 finished with balanced_accuracy value = 0.948331
Fold 3 finished with balanced_accuracy value = 0.959440
Fold 4 finished with balanced_accuracy value = 0.951243
Fold 5 finished with balanced_accuracy value = 0.958813

[I 2026-09-04 02:35:35,180] Trial 70 finished with value: 0.9537061207504532 and parameters: {'nc': 2, 'nb': 6, 'nh': 14, 'ed_nh_ratio': 28, 'ff_hd': 768, 'classifier_hd': 640, 'dropout': 0.32656963931388333, 'lr': 2.515553318839074e-05, 'weight_decay': 0.00989666898085935, 'batch_size': 832, 'patience': 30, 'epochs': 95, 'use_augmentation': True, 'aug_probability': 0.5753057546306296, 'prob': 0.23355891589029282, 'noise_std': 0.2610334486759883, 'dropout_aug': 0.3544084746509544, 'alpha': 0.14249051838399143, 'from_unsupervised': False, 'pretrain_epochs': 30}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.951830
Fold 2 finished with balanced_accuracy value = 0.948311
Fold 3 finished with balanced_accuracy value = 0.951464

[I 2026-09-04 02:39:07,346] Trial 71 pruned.

Fold 1 finished with balanced_accuracy value = 0.950569

[I 2026-09-04 02:40:10,409] Trial 72 pruned.

Fold 1 finished with balanced_accuracy value = 0.952765
Fold 2 finished with balanced_accuracy value = 0.950898
Fold 3 finished with balanced_accuracy value = 0.960070
Fold 4 finished with balanced_accuracy value = 0.954288
Fold 5 finished with balanced_accuracy value = 0.962373

[I 2026-09-04 02:46:05,723] Trial 73 finished with value: 0.9560788098551869 and parameters: {'nc': 2, 'nb': 7, 'nh': 14, 'ed_nh_ratio': 32, 'ff_hd': 768, 'classifier_hd': 1024, 'dropout': 0.2652758991789176, 'lr': 8.150681910574263e-05, 'weight_decay': 0.0032489603439425103, 'batch_size': 1088, 'patience': 30, 'epochs': 120, 'use_augmentation': True, 'aug_probability': 0.6813021806797925, 'prob': 0.1962912085116098, 'noise_std': 0.30927252771388863, 'dropout_aug': 0.30199891737551193, 'alpha': 0.011587760086546071, 'from_unsupervised': True, 'pretrain_epochs': 45}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.952939
Fold 2 finished with balanced_accuracy value = 0.945426
Fold 3 finished with balanced_accuracy value = 0.950445

[I 2026-09-04 02:49:12,444] Trial 74 pruned.

Fold 1 finished with balanced_accuracy value = 0.952303
Fold 2 finished with balanced_accuracy value = 0.945196
Fold 3 finished with balanced_accuracy value = 0.955928

[I 2026-09-04 02:52:54,912] Trial 75 pruned.

Fold 1 finished with balanced_accuracy value = 0.952235
Fold 2 finished with balanced_accuracy value = 0.956260
Fold 3 finished with balanced_accuracy value = 0.961701
Fold 4 finished with balanced_accuracy value = 0.955056
Fold 5 finished with balanced_accuracy value = 0.960791

[I 2026-09-04 02:56:10,535] Trial 76 finished with value: 0.9572085908424907 and parameters: {'nc': 4, 'nb': 5, 'nh': 16, 'ed_nh_ratio': 28, 'ff_hd': 896, 'classifier_hd': 896, 'dropout': 0.2796227414432018, 'lr': 5.195473404523619e-05, 'weight_decay': 0.0024825865926055755, 'batch_size': 1088, 'patience': 25, 'epochs': 110, 'use_augmentation': False, 'aug_probability': 0.3694371975338139, 'prob': 0.15591956648741948, 'noise_std': 0.3991333195830744, 'dropout_aug': 0.35421691732325167, 'alpha': 0.06276438155155631, 'from_unsupervised': False, 'pretrain_epochs': 50}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.950325
Fold 2 finished with balanced_accuracy value = 0.946047
Fold 3 finished with balanced_accuracy value = 0.959341
Fold 4 finished with balanced_accuracy value = 0.952097
Fold 5 finished with balanced_accuracy value = 0.956574

[I 2026-09-04 03:00:08,135] Trial 77 finished with value: 0.9528768078642988 and parameters: {'nc': 6, 'nb': 5, 'nh': 12, 'ed_nh_ratio': 28, 'ff_hd': 1024, 'classifier_hd': 640, 'dropout': 0.2876637567346996, 'lr': 4.585388594284563e-05, 'weight_decay': 0.0014791160191848201, 'batch_size': 1536, 'patience': 25, 'epochs': 95, 'use_augmentation': False, 'aug_probability': 0.21097551835902645, 'prob': 0.11630585253069248, 'noise_std': 0.3781869719390416, 'dropout_aug': 0.13986024562437804, 'alpha': 0.08409081082843853, 'from_unsupervised': False, 'pretrain_epochs': 50}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.953784
Fold 2 finished with balanced_accuracy value = 0.958976
Fold 3 finished with balanced_accuracy value = 0.959796
Fold 4 finished with balanced_accuracy value = 0.956290
Fold 5 finished with balanced_accuracy value = 0.958811

[I 2026-09-04 03:03:51,569] Trial 78 finished with value: 0.9575313929058942 and parameters: {'nc': 4, 'nb': 6, 'nh': 16, 'ed_nh_ratio': 28, 'ff_hd': 896, 'classifier_hd': 1024, 'dropout': 0.2634522973789175, 'lr': 5.180638218424004e-05, 'weight_decay': 0.0037572167334358215, 'batch_size': 960, 'patience': 25, 'epochs': 110, 'use_augmentation': False, 'aug_probability': 0.3656499053602632, 'prob': 0.15244794314585283, 'noise_std': 0.08757356092684229, 'dropout_aug': 0.3545797102558115, 'alpha': 0.10179640114586277, 'from_unsupervised': False, 'pretrain_epochs': 55}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.950105

[I 2026-09-04 03:04:43,192] Trial 79 pruned.

Fold 1 finished with balanced_accuracy value = 0.946648
Fold 2 finished with balanced_accuracy value = 0.956650
Fold 3 finished with balanced_accuracy value = 0.955324

[I 2026-09-04 03:05:49,083] Trial 80 pruned.

Fold 1 finished with balanced_accuracy value = 0.950340
Fold 2 finished with balanced_accuracy value = 0.955315
Fold 3 finished with balanced_accuracy value = 0.955982

[I 2026-09-04 03:07:45,995] Trial 81 pruned.

Fold 1 finished with balanced_accuracy value = 0.951627
Fold 2 finished with balanced_accuracy value = 0.955803
Fold 3 finished with balanced_accuracy value = 0.956879

[I 2026-09-04 03:09:43,461] Trial 82 pruned.

Fold 1 finished with balanced_accuracy value = 0.953650
Fold 2 finished with balanced_accuracy value = 0.952234
Fold 3 finished with balanced_accuracy value = 0.950784

[I 2026-09-04 03:12:21,343] Trial 83 pruned.

Fold 1 finished with balanced_accuracy value = 0.951783

[I 2026-09-04 03:13:08,181] Trial 84 pruned.

Fold 1 finished with balanced_accuracy value = 0.952644
Fold 2 finished with balanced_accuracy value = 0.958385
Fold 3 finished with balanced_accuracy value = 0.953460

[I 2026-09-04 03:14:01,280] Trial 85 pruned.

Fold 1 finished with balanced_accuracy value = 0.954020
Fold 2 finished with balanced_accuracy value = 0.952424
Fold 3 finished with balanced_accuracy value = 0.960461
Fold 4 finished with balanced_accuracy value = 0.952529
Fold 5 finished with balanced_accuracy value = 0.962624

[I 2026-09-04 03:17:08,932] Trial 86 finished with value: 0.9564115950452766 and parameters: {'nc': 6, 'nb': 5, 'nh': 16, 'ed_nh_ratio': 24, 'ff_hd': 128, 'classifier_hd': 768, 'dropout': 0.2618562233803836, 'lr': 0.00013295967839574538, 'weight_decay': 0.0044051285414993, 'batch_size': 1408, 'patience': 25, 'epochs': 195, 'use_augmentation': False, 'aug_probability': 0.7232950327127972, 'prob': 0.11267546657699531, 'noise_std': 0.3709139241866883, 'dropout_aug': 0.11080751804047587, 'alpha': 0.04213172519156471, 'from_unsupervised': False, 'pretrain_epochs': 50}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.953110
Fold 2 finished with balanced_accuracy value = 0.952027
Fold 3 finished with balanced_accuracy value = 0.958451

[I 2026-09-04 03:18:52,932] Trial 87 pruned.

Fold 1 finished with balanced_accuracy value = 0.947924

[I 2026-09-04 03:19:37,660] Trial 88 pruned.

Fold 1 finished with balanced_accuracy value = 0.942760

[I 2026-09-04 03:20:25,322] Trial 89 pruned.

Fold 1 finished with balanced_accuracy value = 0.948967

[I 2026-09-04 03:21:00,890] Trial 90 pruned.

Fold 1 finished with balanced_accuracy value = 0.955484
Fold 2 finished with balanced_accuracy value = 0.951769
Fold 3 finished with balanced_accuracy value = 0.959126

[I 2026-09-04 03:24:13,558] Trial 91 pruned.

Fold 1 finished with balanced_accuracy value = 0.952596
Fold 2 finished with balanced_accuracy value = 0.956433
Fold 3 finished with balanced_accuracy value = 0.958989

[I 2026-09-04 03:27:52,432] Trial 92 pruned.

Fold 1 finished with balanced_accuracy value = 0.955731
Fold 2 finished with balanced_accuracy value = 0.959548
Fold 3 finished with balanced_accuracy value = 0.959018

[I 2026-09-04 03:29:35,868] Trial 93 pruned.

Fold 1 finished with balanced_accuracy value = 0.952983
Fold 2 finished with balanced_accuracy value = 0.958827
Fold 3 finished with balanced_accuracy value = 0.964869
Fold 4 finished with balanced_accuracy value = 0.957613
Fold 5 finished with balanced_accuracy value = 0.964708

[I 2026-09-04 03:35:56,781] Trial 94 finished with value: 0.9597999526642486 and parameters: {'nc': 12, 'nb': 5, 'nh': 16, 'ed_nh_ratio': 32, 'ff_hd': 768, 'classifier_hd': 896, 'dropout': 0.2687257882171326, 'lr': 6.657892559003578e-05, 'weight_decay': 0.00182636810911123, 'batch_size': 1088, 'patience': 30, 'epochs': 120, 'use_augmentation': True, 'aug_probability': 0.4855436028429998, 'prob': 0.24825793776512686, 'noise_std': 0.09298600507618181, 'dropout_aug': 0.3761574179552977, 'alpha': 0.05060301781572779, 'from_unsupervised': False, 'pretrain_epochs': 40}. Best is trial 24 with value: 0.9615867721230634.

Fold 1 finished with balanced_accuracy value = 0.950989
Fold 2 finished with balanced_accuracy value = 0.943760
Fold 3 finished with balanced_accuracy value = 0.958568

[I 2026-09-04 03:41:57,983] Trial 95 pruned.

Fold 1 finished with balanced_accuracy value = 0.948905

[I 2026-09-04 03:43:09,628] Trial 96 pruned.

Fold 1 finished with balanced_accuracy value = 0.952963
Fold 2 finished with balanced_accuracy value = 0.959828
Fold 3 finished with balanced_accuracy value = 0.957390

[I 2026-09-04 03:47:54,056] Trial 97 pruned.

Fold 1 finished with balanced_accuracy value = 0.950458

[I 2026-09-04 03:49:40,621] Trial 98 pruned.

Fold 1 finished with balanced_accuracy value = 0.950296

[I 2026-09-04 03:50:45,741] Trial 99 pruned.

Best value (balanced_accuracy) = 0.9615867721230634
Best hyperparameters saved to: model_scadam_hp_tune/best_params.txt

[21]:

{'nc': 4,
 'nb': 5,
 'nh': 16,
 'ed_nh_ratio': 28,
 'ff_hd': 768,
 'classifier_hd': 896,
 'dropout': 0.23698142213184742,
 'lr': 6.963690736722135e-05,
 'weight_decay': 0.0034732362400466494,
 'batch_size': 1408,
 'patience': 30,
 'epochs': 195,
 'use_augmentation': True,
 'aug_probability': 0.7677578787467152,
 'prob': 0.20685827234760135,
 'noise_std': 0.10838449541841462,
 'dropout_aug': 0.00177856974742846,
 'alpha': 0.04645915647446934,
 'from_unsupervised': False,
 'pretrain_epochs': 25}

[22]:

# Train model using optimal parameters from model_scadam_hp_tune folder
scparadise.scadam.train_tuned(
    adata_train_1,
    path='', # path to save model
    path_tuned='model_scadam_hp_tune', # path to a folder with tuned hyperparameters
    model_name='model_scadam_tuned', # folder name with model
    celltype_keys=['cell_type'],
    eval_metric=['accuracy', 'balanced_accuracy']
)

Successfully loaded tuned hyperparameters!
Device: cuda
Number of features: 1200
Label hierarchy: cell_type
Annotation levels weights using strategy 'linear_offset':
  cell_type: 15 cell types, 1.0 relative weight

Dataset split:
Train dataset contains: 13738 cells, it is 80.0 % of input dataset
Validation dataset contains: 3435 cells, it is 20.0 % of input dataset

Training scAdam model:  37%|██████▋           | 73/195 [00:41<01:09,  1.75it/s]

Early stopping triggered! Best score: 0.9635
Training completed!
Fitting unknown cells detector



UnknownCellDetector fitted successfully!
Model saved to model_scadam_tuned

[23]:

# Predict cell types using trained model
adata_test = scparadise.scadam.predict(
    adata_test,
    path_model = 'model_scadam_tuned'
)

Cell type annotation using the 2nd version of the scAdam model
scAdam model with unknown detector loaded from model_scadam_tuned
Gene alignment:
  Model features: 1200
  Matched features: 1200 (100.0%)

Predicting: 100%|█████████████████████████████| 72/72 [00:00<00:00, 161.81it/s]

Added cell type column: pred_celltype_l1
Added probabilities column: pred_celltype_l1_probability

Evaluation of prediction results#

Hyperparameters tuning leads to an increase in the model’s overall accuracy (+4.37%) and balanced accuracy (+1.33%).

Below are examples of improved model performance for certain cell types.

  1. CD8-positive, alpha-beta T cell (precision - 0.8742 vs 0.7178)

  2. B cell (precision - 0.9258 vs 0.6606)

  3. epithelial cell (sensitivity - 0.6108 vs 0.5229)

  4. natural killer cell (precision - 0.6977 vs 0.5815)

  5. regulatory T cell (precision - 0.7010 vs 0.5116)

However, a decrease in prediction accuracy may be observed for some cell types.

  1. CD4-positive, alpha-beta T cell (sensitivity - 0.3840 vs 0.4553)

[24]:

## Check model quality
df_hp_tuned = scparadise.scnoah.report_classif_full(
    adata_test,
    celltype='cell_type',
    pred_celltype='pred_celltype_l1'
)
df_hp_tuned

[24]:
precision recall/sensitivity specificity f1-score geometric mean index balanced accuracy number of cells
B cell 0.9258 0.98 0.9901 0.9521 0.985 0.9693 2050
CD4-positive, alpha-beta T cell 0.3840 0.8578 0.8723 0.5305 0.865 0.7472 1554
CD8-positive, alpha-beta T cell 0.8742 0.9133 0.9862 0.8933 0.949 0.8941 1742
endothelial cell 0.9934 0.9869 0.9998 0.9901 0.9933 0.9854 457
endothelial cell of lymphatic vessel 1.0000 0.9808 1.0 0.9903 0.9903 0.9789 52
epithelial cell 0.9957 0.6108 0.9981 0.7571 0.7808 0.586 7505
fibroblast 0.9820 0.966 0.9993 0.9739 0.9825 0.9621 676
mast cell 0.9726 0.9595 0.9998 0.966 0.9794 0.9554 148
mature alpha-beta T cell 0.0941 0.8933 0.9646 0.1703 0.9283 0.8556 75
mural cell 0.9449 0.9817 0.9986 0.963 0.9901 0.9787 437
myeloid cell 0.9782 0.9876 0.9991 0.9829 0.9933 0.9856 726
natural killer cell 0.6977 0.9731 0.9948 0.8127 0.9839 0.9659 223
plasma cell 0.9825 0.9974 0.9979 0.9899 0.9977 0.9953 1919
plasmacytoid dendritic cell 0.9423 0.98 0.9998 0.9608 0.9899 0.9779 50
regulatory T cell 0.7010 0.7245 0.988 0.7125 0.846 0.6969 686
macro avg 0.8312 0.9195 0.9859 0.843 0.9503 0.9023
weighted avg 0.9018 0.8035 0.985 0.8243 0.8841 0.7791
Accuracy 0.8035
Balanced accuracy 0.9195 
[25]:

# Order cell type colors
celltype = np.unique(adata_test.obs['cell_type']).tolist()
adata_test.obs['cell_type'] = pd.Categorical(
    values=adata_test.obs['cell_type'], categories=celltype, ordered=True
)
adata_test.obs['pred_celltype_l1'] = pd.Categorical(
    values=adata_test.obs['pred_celltype_l1'], categories=celltype, ordered=True
)
# Add prediction status. Label cells as correct or incorrect based on the comparison between ground truth cell types and predictions.
scparadise.scnoah.pred_status(adata_test, celltype='cell_type', pred_celltype='pred_celltype_l1')

[26]:

# Visualise predicted cell types levels, prediction probabilities and prediction status
sc.pl.umap(
    adata_test,
    color=[
        'cell_type',
        'pred_celltype_l1',
        'pred_celltype_l1_probability',
        'pred_status'
    ],
    frameon = False,
    add_outline = True,
    legend_loc = 'right margin',
    legend_fontsize = 7,
    legend_fontoutline = 1,
    ncols=2,
    wspace = 0.7,
    hspace = 0.1
)
../../../_images/tutorials_notebooks_scAdam_scAdam_model_optimization_37_0.png 
[27]:

# Compare prediction results
# 'untuned' row represents untuned model
# 'hp tuned' row represents tuned model
df.compare(df_hp_tuned, keep_equal=True, align_axis = 0, result_names=('untuned', 'hp tuned'))

[27]:
precision recall/sensitivity specificity f1-score geometric mean index balanced accuracy
B cell untuned 0.6606 0.982 0.9364 0.7899 0.9589 0.9237
hp tuned 0.9258 0.98 0.9901 0.9521 0.985 0.9693
CD4-positive, alpha-beta T cell untuned 0.4553 0.816 0.9094 0.5845 0.8614 0.7351
hp tuned 0.3840 0.8578 0.8723 0.5305 0.865 0.7472
CD8-positive, alpha-beta T cell untuned 0.7178 0.876 0.9638 0.789 0.9188 0.8369
hp tuned 0.8742 0.9133 0.9862 0.8933 0.949 0.8941
endothelial cell untuned 1.0000 0.9912 1.0 0.9956 0.9956 0.9904
hp tuned 0.9934 0.9869 0.9998 0.9901 0.9933 0.9854
endothelial cell of lymphatic vessel untuned 1.0000 0.9615 1.0 0.9804 0.9806 0.9578
hp tuned 1.0000 0.9808 1.0 0.9903 0.9903 0.9789
epithelial cell untuned 0.9975 0.5229 0.9991 0.6861 0.7227 0.4975
hp tuned 0.9957 0.6108 0.9981 0.7571 0.7808 0.586
fibroblast untuned 0.9864 0.9689 0.9995 0.9776 0.9841 0.9655
hp tuned 0.9820 0.966 0.9993 0.9739 0.9825 0.9621
mast cell untuned 0.9793 0.9595 0.9998 0.9693 0.9794 0.9554
hp tuned 0.9726 0.9595 0.9998 0.966 0.9794 0.9554
mature alpha-beta T cell untuned 0.1199 0.9067 0.9726 0.2118 0.9391 0.876
hp tuned 0.0941 0.8933 0.9646 0.1703 0.9283 0.8556
mural cell untuned 0.9554 0.9794 0.9989 0.9672 0.9891 0.9764
hp tuned 0.9449 0.9817 0.9986 0.963 0.9901 0.9787
myeloid cell untuned 0.9781 0.9862 0.9991 0.9822 0.9926 0.9841
hp tuned 0.9782 0.9876 0.9991 0.9829 0.9933 0.9856
natural killer cell untuned 0.5815 0.9596 0.9915 0.7242 0.9754 0.9484
hp tuned 0.6977 0.9731 0.9948 0.8127 0.9839 0.9659
plasma cell untuned 0.9647 0.9979 0.9957 0.981 0.9968 0.9939
hp tuned 0.9825 0.9974 0.9979 0.9899 0.9977 0.9953
plasmacytoid dendritic cell untuned 0.9800 0.98 0.9999 0.98 0.9899 0.978
hp tuned 0.9423 0.98 0.9998 0.9608 0.9899 0.9779
regulatory T cell untuned 0.5116 0.7055 0.9738 0.5931 0.8289 0.6686
hp tuned 0.7010 0.7245 0.988 0.7125 0.846 0.6969
macro avg untuned 0.7926 0.9062 0.9826 0.8141 0.9409 0.8858
hp tuned 0.8312 0.9195 0.9859 0.843 0.9503 0.9023
weighted avg untuned 0.8545 0.7598 0.9796 0.7657 0.8534 0.7299
hp tuned 0.9018 0.8035 0.985 0.8243 0.8841 0.7791
Accuracy untuned 0.7598
hp tuned 0.8035
Balanced accuracy untuned 0.9062
hp tuned 0.9195 
[28]:

# Compare prediction results
# 'warm start' row represents warm start trained model
# 'hp tuned' row represents tuned model
df_warm_start.compare(df_hp_tuned, keep_equal=True, align_axis = 0, result_names=('warm start', 'hp tuned'))

[28]:
precision recall/sensitivity specificity f1-score geometric mean index balanced accuracy
B cell warm start 0.9906 0.9771 0.9988 0.9838 0.9879 0.9738
hp tuned 0.9258 0.98 0.9901 0.9521 0.985 0.9693
CD4-positive, alpha-beta T cell warm start 0.8309 0.8571 0.9838 0.8438 0.9183 0.8326
hp tuned 0.3840 0.8578 0.8723 0.5305 0.865 0.7472
CD8-positive, alpha-beta T cell warm start 0.9252 0.9156 0.9922 0.9204 0.9531 0.9015
hp tuned 0.8742 0.9133 0.9862 0.8933 0.949 0.8941
endothelial cell warm start 0.9956 0.9934 0.9999 0.9945 0.9967 0.9927
hp tuned 0.9934 0.9869 0.9998 0.9901 0.9933 0.9854
endothelial cell of lymphatic vessel warm start 1.0000 1.0 1.0 1.0 1.0 1.0
hp tuned 1.0000 0.9808 1.0 0.9903 0.9903 0.9789
epithelial cell warm start 0.9930 0.98 0.9952 0.9865 0.9876 0.9738
hp tuned 0.9957 0.6108 0.9981 0.7571 0.7808 0.586
fibroblast warm start 0.9938 0.9556 0.9998 0.9744 0.9774 0.9512
hp tuned 0.9820 0.966 0.9993 0.9739 0.9825 0.9621
mast cell warm start 0.9931 0.973 0.9999 0.9829 0.9864 0.9703
hp tuned 0.9726 0.9595 0.9998 0.966 0.9794 0.9554
mature alpha-beta T cell warm start 0.6837 0.8933 0.9983 0.7746 0.9444 0.8825
hp tuned 0.0941 0.8933 0.9646 0.1703 0.9283 0.8556
mural cell warm start 0.9272 0.9908 0.9981 0.958 0.9945 0.9882
hp tuned 0.9449 0.9817 0.9986 0.963 0.9901 0.9787
myeloid cell warm start 0.9876 0.9848 0.9995 0.9862 0.9921 0.9829
hp tuned 0.9782 0.9876 0.9991 0.9829 0.9933 0.9856
natural killer cell warm start 0.8220 0.9731 0.9974 0.8912 0.9852 0.9682
hp tuned 0.6977 0.9731 0.9948 0.8127 0.9839 0.9659
plasma cell warm start 0.9912 0.9958 0.999 0.9935 0.9974 0.9945
hp tuned 0.9825 0.9974 0.9979 0.9899 0.9977 0.9953
plasmacytoid dendritic cell warm start 0.9608 0.98 0.9999 0.9703 0.9899 0.9779
hp tuned 0.9423 0.98 0.9998 0.9608 0.9899 0.9779
regulatory T cell warm start 0.7475 0.7726 0.9898 0.7599 0.8745 0.7481
hp tuned 0.7010 0.7245 0.988 0.7125 0.846 0.6969
macro avg warm start 0.9228 0.9495 0.9968 0.9347 0.9724 0.9426
hp tuned 0.8312 0.9195 0.9859 0.843 0.9503 0.9023
weighted avg warm start 0.9580 0.9564 0.9952 0.957 0.9753 0.9486
hp tuned 0.9018 0.8035 0.985 0.8243 0.8841 0.7791
Accuracy warm start 0.9564
hp tuned 0.8035
Balanced accuracy warm start 0.9495
hp tuned 0.9195

Recommendation#

When possible, we recommend warm-start training rather than hyperparameter tuning. If an additional training dataset is unavailable, hyperparameter tuning can be used to optimize model performance.

Warm-start approaches can reuse information from prior training or tuning runs, potentially reducing the number of new experiments needed; however, their benefit depends on how similar the new training data are to the original data.

[29]:

import session_info2
session_info2.session_info()

[29]:
PackageVersion
pandas2.3.3
scanpy1.11.5
scparadise1.2.0
matplotlib3.11.0
numpy2.4.6
anndata0.12.16
ComponentInfo
Python3.11.15 (main, Jun 11 2026, 15:20:16) [GCC 14.3.0]
OSLinux-6.8.0-138-generic-x86_64-with-glibc2.39
Updated2026-09-04 03:51
Dependencies
DependencyVersion
joblib1.5.3
scikit-image0.26.0
pycparser3.0 (3.00)
PyYAML6.0.3
alembic1.18.4
rich15.0.0
lazy-loader0.5
jedi0.20.0
setuptools81.0.0
jupyter_client8.9.1
google-crc32c1.8.0
donfig0.8.1.post1
pytz2026.2
ipykernel7.3.0
traitlets5.15.1
fsspec2026.4.0
pyzmq27.1.0
executing2.2.1
jupyter_core5.9.1
torch2.12.0 (2.12.0+cu130)
numcodecs0.16.5
h5py3.16.0
leidenalg0.12.0
matplotlib-inline0.2.2
ipython9.14.1
cffi2.0.0
igraph1.0.0
psutil7.2.2
scverse-misc0.0.3
pynndescent0.6.0
asttokens3.0.1
charset-normalizer3.4.7
umap-learn0.5.12
comm0.2.3
Cython3.3.0
legacy-api-wrap1.5
typing3.7.4.3
platformdirs4.10.0
dataclasses0.6
pytorch-tabnet4.1.0
kiwisolver1.5.0
numba0.65.1
six1.17.0
tornado6.5.7
scikit-learn1.9.0
pyarrow24.0.0
SQLAlchemy2.0.50
llvmlite0.47.0
imbalanced-learn0.14.2
slicer0.0.8
fonttools4.63.0
sklearn-compat0.1.6
mudata0.3.8
debugpy1.8.21
triton3.7.0
opt_einsum3.4.0
texttable1.7.0
defusedxml0.7.1
zarr3.1.6
python-dateutil2.9.0.post0
typing_extensions4.15.0
scikit-misc0.5.2
cloudpickle3.1.2
MarkupSafe3.0.3
Pygments2.20.0
parso0.8.7
muon0.1.7
xarray2026.4.0
session-info20.4.1
sympy1.14.0
argparse1.4.0 (1.1)
cycler0.12.1
optuna4.9.0
Mako1.3.12
packaging26.0
colorlog6.10.1
stack-data0.6.3
prompt_toolkit3.0.52
pillow12.2.0
scipy1.17.1
sparse0.18.0
decorator5.3.1
narwhals2.22.1
wcwidth0.8.1
patsy1.0.2
greenlet3.5.1
statsmodels0.14.6
tqdm4.68.2
seaborn0.13.2
pure_eval0.2.3
cuda-pathfinder1.5.5
mpmath1.3.0
cuda-bindings13.3.1
threadpoolctl3.6.0
natsort8.4.0
pyparsing3.3.2
plottable0.1.5
shap0.49.1
pathlib1.0.1
Copyable Markdown 
| Package    | Version |
| ---------- | ------- |
| pandas     | 2.3.3   |
| scanpy     | 1.11.5  |
| scparadise | 1.2.0   |
| matplotlib | 3.11.0  |
| numpy      | 2.4.6   |
| anndata    | 0.12.16 |

| Dependency         | Version               |
| ------------------ | --------------------- |
| joblib             | 1.5.3                 |
| scikit-image       | 0.26.0                |
| pycparser          | 3.0 (3.00)            |
| PyYAML             | 6.0.3                 |
| alembic            | 1.18.4                |
| rich               | 15.0.0                |
| lazy-loader        | 0.5                   |
| jedi               | 0.20.0                |
| setuptools         | 81.0.0                |
| jupyter_client     | 8.9.1                 |
| google-crc32c      | 1.8.0                 |
| donfig             | 0.8.1.post1           |
| pytz               | 2026.2                |
| ipykernel          | 7.3.0                 |
| traitlets          | 5.15.1                |
| fsspec             | 2026.4.0              |
| pyzmq              | 27.1.0                |
| executing          | 2.2.1                 |
| jupyter_core       | 5.9.1                 |
| torch              | 2.12.0 (2.12.0+cu130) |
| numcodecs          | 0.16.5                |
| h5py               | 3.16.0                |
| leidenalg          | 0.12.0                |
| matplotlib-inline  | 0.2.2                 |
| ipython            | 9.14.1                |
| cffi               | 2.0.0                 |
| igraph             | 1.0.0                 |
| psutil             | 7.2.2                 |
| scverse-misc       | 0.0.3                 |
| pynndescent        | 0.6.0                 |
| asttokens          | 3.0.1                 |
| charset-normalizer | 3.4.7                 |
| umap-learn         | 0.5.12                |
| comm               | 0.2.3                 |
| Cython             | 3.3.0                 |
| legacy-api-wrap    | 1.5                   |
| typing             | 3.7.4.3               |
| platformdirs       | 4.10.0                |
| dataclasses        | 0.6                   |
| pytorch-tabnet     | 4.1.0                 |
| kiwisolver         | 1.5.0                 |
| numba              | 0.65.1                |
| six                | 1.17.0                |
| tornado            | 6.5.7                 |
| scikit-learn       | 1.9.0                 |
| pyarrow            | 24.0.0                |
| SQLAlchemy         | 2.0.50                |
| llvmlite           | 0.47.0                |
| imbalanced-learn   | 0.14.2                |
| slicer             | 0.0.8                 |
| fonttools          | 4.63.0                |
| sklearn-compat     | 0.1.6                 |
| mudata             | 0.3.8                 |
| debugpy            | 1.8.21                |
| triton             | 3.7.0                 |
| opt_einsum         | 3.4.0                 |
| texttable          | 1.7.0                 |
| defusedxml         | 0.7.1                 |
| zarr               | 3.1.6                 |
| python-dateutil    | 2.9.0.post0           |
| typing_extensions  | 4.15.0                |
| scikit-misc        | 0.5.2                 |
| cloudpickle        | 3.1.2                 |
| MarkupSafe         | 3.0.3                 |
| Pygments           | 2.20.0                |
| parso              | 0.8.7                 |
| muon               | 0.1.7                 |
| xarray             | 2026.4.0              |
| session-info2      | 0.4.1                 |
| sympy              | 1.14.0                |
| argparse           | 1.4.0 (1.1)           |
| cycler             | 0.12.1                |
| optuna             | 4.9.0                 |
| Mako               | 1.3.12                |
| packaging          | 26.0                  |
| colorlog           | 6.10.1                |
| stack-data         | 0.6.3                 |
| prompt_toolkit     | 3.0.52                |
| pillow             | 12.2.0                |
| scipy              | 1.17.1                |
| sparse             | 0.18.0                |
| decorator          | 5.3.1                 |
| narwhals           | 2.22.1                |
| wcwidth            | 0.8.1                 |
| patsy              | 1.0.2                 |
| greenlet           | 3.5.1                 |
| statsmodels        | 0.14.6                |
| tqdm               | 4.68.2                |
| seaborn            | 0.13.2                |
| pure_eval          | 0.2.3                 |
| cuda-pathfinder    | 1.5.5                 |
| mpmath             | 1.3.0                 |
| cuda-bindings      | 13.3.1                |
| threadpoolctl      | 3.6.0                 |
| natsort            | 8.4.0                 |
| pyparsing          | 3.3.2                 |
| plottable          | 0.1.5                 |
| shap               | 0.49.1                |
| pathlib            | 1.0.1                 |

| Component | Info                                               |
| --------- | -------------------------------------------------- |
| Python    | 3.11.15 (main, Jun 11 2026, 15:20:16) [GCC 14.3.0] |
| OS        | Linux-6.8.0-138-generic-x86_64-with-glibc2.39      |
| Updated   | 2026-09-04 03:51                                   |