BEN_tutorial_Ⅱ_cross_modality
Note: to replicate experiments conveniently, we did not package key functions as ‘.exe’ files here. Preprocessing is also simplified.
# 1.0 Set up
## 1.1 Verify Runtime Settings
** IMPORTANT **
In the “Runtime” menu for the notebook window, select “Change runtime type.” Ensure that the following are selected: * Runtime Type = Python 3 * Hardware Accelerator = GPU
See your Hardware Accelerator in Edit/Notebook settings in Colab Menu(On the upper left usually).
## 1.2 Set up environment and install all necessary packages
After installation, you might ** need to restart ** the runtime in order to use newly installed versions (Tensorflow 1.15).
[ ]:
!pip install SimpleITK tensorflow-gpu==1.15.4 keras==2.2.4 h5py==2.10 scikit-image==0.16.2
1.3 Download code from github.
[2]:
!git clone https://github.com/yu02019/BEN.git
Cloning into 'BEN'...
remote: Enumerating objects: 162, done.
remote: Counting objects: 100% (19/19), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 162 (delta 2), reused 10 (delta 0), pack-reused 143
Receiving objects: 100% (162/162), 6.47 MiB | 14.47 MiB/s, done.
Resolving deltas: 100% (46/46), done.
[3]:
cd BEN
/content/BEN
## 1.4 Download MRI data.
[4]:
!gdown --id 1fmyBtxNJAYFtlwVBY5IiFsmUixc2MsU9
!sh download.sh
/usr/local/lib/python3.7/dist-packages/gdown/cli.py:131: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.
category=FutureWarning,
Downloading...
From: https://drive.google.com/uc?id=1fmyBtxNJAYFtlwVBY5IiFsmUixc2MsU9
To: /content/BEN/download.sh
100% 273/273 [00:00<00:00, 512kB/s]
/usr/local/lib/python3.7/dist-packages/gdown/cli.py:131: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.
category=FutureWarning,
Downloading...
From: https://drive.google.com/uc?id=1z-VMfPvFILNcaEc9jSYzcnub7t4ZcrfJ
To: /content/BEN/cross_domain/cross_domain.zip
100% 26.8M/26.8M [00:00<00:00, 97.0MB/s]
Archive: cross_domain.zip
creating: 7T/
creating: 7T/src/
inflating: 7T/src/20171226_185416T1RAREs30001a001.nii.gz
inflating: 7T/src/20180321_133106T2TurboRAREs20001a001.nii.gz
inflating: 7T/src/20180510_143551T1RAREs30001a001.nii.gz
creating: epi/
creating: epi/src/
inflating: epi/src/ASD_Xinhua_SJL_20191215_con06__E3_P1.nii.gz
inflating: epi/src/ASD_Xinhua_SJL_20191215_tcell_0__E3_P1.1.nii.gz
creating: rat/
creating: rat/label/
inflating: rat/label/42d_cas_27_T2.nii.gz
creating: rat/src/
inflating: rat/src/42d_cas_27_T2.nii.gz
inflating: rat/src/42d_cas_40_T2.nii.gz
inflating: rat/src/42d_cas_44_T2.nii.gz
/usr/local/lib/python3.7/dist-packages/gdown/cli.py:131: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.
category=FutureWarning,
Downloading...
From: https://drive.google.com/uc?id=1K5qIe6o8uG7cw_Uu0H3aB0FE40YXb9Hp
To: /content/BEN/weight/weight.zip
100% 69.3M/69.3M [00:00<00:00, 202MB/s]
Archive: weight.zip
creating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/.data-00000-of-00002
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/.data-00001-of-00002
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/.index
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/checkpoint
creating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012056/
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012056/.data-00000-of-00002
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012056/.data-00001-of-00002
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012056/.index
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012056/checkpoint
The MRI data look like:
(src: raw MRI scans; label:ground truth)
[5]:
import os
import logging
import warnings
import tensorflow as tf
warnings.filterwarnings("ignore")
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' # tf log errors only
logging.getLogger('tensorflow').setLevel(logging.ERROR)
print(tf.__version__)
1.15.4
# 2. Cross modality (T2WI -> EPI)
## 2.1 Declare key function and path In this case, no label required.
[6]:
from utils.update_model import update_weight
# config = tf.ConfigProto()
# config.gpu_options.allow_growth = True
# tf.Session(config=config)
input_folder = r'cross_domain/epi/src'
label_folder = '' # No label required.
output_folder_base = r'cross_domain/epi' # save pred nii files.(time id based on weight)
old_model_name = r'weight/unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/' # load and adjust BN base on this old model(fixed/source domain usually)
new_model_name = 'EPI-DA' # The prefix of new weight filename will use this string
cvs_result_base = r'cross_domain/epi' # evaluation metric in csv. (path of csv, time id)
need_rotate = True
## 2.2 Run BEN’s domain transfer/adaptation
[7]:
# transfer BEN to new domain/dataset
new_weight = update_weight(input_folder, label_folder, need_mkdir=True, weight=old_model_name,
need_rotate=need_rotate, model_name=new_model_name,
BN_list=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1], freeze=True)
print('Finetune done.\n New model path is : ', new_weight)
# setting filename
import re
num_extract = re.findall(r'\d+', new_weight)
output_folder = output_folder_base + '/' + 'pred' + '-' + num_extract[0] # e.g. 'pred-04061444'
DA_path = output_folder
cvs_result = cvs_result_base + '/' + new_weight.replace('weight/', '').replace('/', '') + '.csv'
new_weight = new_weight + '.hdf5'
from utils.inference import inference_pipeline
import tensorflow.keras.backend as K
K.clear_session() # release GRAM
# run inference
inference_pipeline(input_folder,
output_folder, is_mkdir=True,
weight=new_weight,
need_rotate=need_rotate,
BN_list=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
)
Finding 2 nii.gz format files.
Reading: ASD_Xinhua_SJL_20191215_con06__E3_P1.nii.gz
(35, 90, 90)
Reading: ASD_Xinhua_SJL_20191215_tcell_0__E3_P1.1.nii.gz
(35, 90, 90)
Done.
Create empty label matrix!
********** Inferring CT/MRI scans: **********
Trainable lay: batch_normalization
Trainable lay: batch_normalization_1
Trainable lay: batch_normalization_2
Trainable lay: batch_normalization_3
Trainable lay: batch_normalization_4
Trainable lay: batch_normalization_5
Trainable lay: batch_normalization_6
Trainable lay: batch_normalization_7
Trainable lay: batch_normalization_8
Trainable lay: batch_normalization_9
Only finetune BN on target domain!
Using TensorFlow backend.
Train on 63 samples, validate on 7 samples
Epoch 1/30
63/63 [==============================] - 8s 125ms/sample - loss: 3.9280 - dice_coef: 5.8535e-09 - val_loss: 0.2037 - val_dice_coef: 5.8334e-07
Epoch 2/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9304 - dice_coef: 5.8434e-09 - val_loss: 0.2103 - val_dice_coef: 2.9243e-07
Epoch 3/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9269 - dice_coef: 5.8409e-09 - val_loss: 0.3412 - val_dice_coef: 1.5403e-07
Epoch 4/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9328 - dice_coef: 5.8362e-09 - val_loss: 0.5299 - val_dice_coef: 1.0724e-07
Epoch 5/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9201 - dice_coef: 5.8169e-09 - val_loss: 0.6718 - val_dice_coef: 9.2216e-08
Epoch 6/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9290 - dice_coef: 5.8479e-09 - val_loss: 0.7151 - val_dice_coef: 8.6451e-08
Epoch 7/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9314 - dice_coef: 5.8383e-09 - val_loss: 0.7327 - val_dice_coef: 8.3428e-08
Epoch 8/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9308 - dice_coef: 5.8480e-09 - val_loss: 0.7460 - val_dice_coef: 8.0850e-08
Epoch 9/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9233 - dice_coef: 5.7889e-09 - val_loss: 0.7516 - val_dice_coef: 8.0037e-08
Epoch 10/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9313 - dice_coef: 5.8513e-09 - val_loss: 0.7566 - val_dice_coef: 7.8941e-08
Epoch 11/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9303 - dice_coef: 5.8468e-09 - val_loss: 0.7610 - val_dice_coef: 7.7833e-08
Epoch 12/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9263 - dice_coef: 5.7809e-09 - val_loss: 0.7677 - val_dice_coef: 7.6665e-08
Epoch 13/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9325 - dice_coef: 5.8102e-09 - val_loss: 0.7725 - val_dice_coef: 7.6264e-08
Epoch 14/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9300 - dice_coef: 5.8392e-09 - val_loss: 0.7716 - val_dice_coef: 7.6227e-08
Epoch 15/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9324 - dice_coef: 5.8385e-09 - val_loss: 0.7708 - val_dice_coef: 7.6240e-08
Epoch 16/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9271 - dice_coef: 5.8170e-09 - val_loss: 0.7724 - val_dice_coef: 7.5868e-08
Epoch 17/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9310 - dice_coef: 5.8462e-09 - val_loss: 0.7693 - val_dice_coef: 7.6263e-08
Epoch 18/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9304 - dice_coef: 5.8440e-09 - val_loss: 0.7670 - val_dice_coef: 7.6669e-08
Epoch 19/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9306 - dice_coef: 5.8281e-09 - val_loss: 0.7665 - val_dice_coef: 7.7074e-08
Epoch 20/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9283 - dice_coef: 5.8528e-09 - val_loss: 0.7649 - val_dice_coef: 7.7115e-08
Epoch 21/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9325 - dice_coef: 5.8536e-09 - val_loss: 0.7635 - val_dice_coef: 7.7141e-08
Epoch 22/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9299 - dice_coef: 5.8502e-09 - val_loss: 0.7625 - val_dice_coef: 7.7190e-08
Epoch 23/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9288 - dice_coef: 5.8378e-09 - val_loss: 0.7615 - val_dice_coef: 7.7274e-08
Epoch 24/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9241 - dice_coef: 5.8052e-09 - val_loss: 0.7627 - val_dice_coef: 7.7412e-08
Epoch 25/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9261 - dice_coef: 5.8347e-09 - val_loss: 0.7666 - val_dice_coef: 7.6611e-08
Epoch 26/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9357 - dice_coef: 5.8460e-09 - val_loss: 0.7659 - val_dice_coef: 7.6444e-08
Epoch 27/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9296 - dice_coef: 5.8129e-09 - val_loss: 0.7654 - val_dice_coef: 7.6967e-08
Epoch 28/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9263 - dice_coef: 5.7834e-09 - val_loss: 0.7746 - val_dice_coef: 7.6036e-08
Epoch 29/30
63/63 [==============================] - 0s 3ms/sample - loss: 3.9174 - dice_coef: 5.8142e-09 - val_loss: 0.7790 - val_dice_coef: 7.5339e-08
Epoch 30/30
63/63 [==============================] - 0s 4ms/sample - loss: 3.9228 - dice_coef: 5.7944e-09 - val_loss: 0.7777 - val_dice_coef: 7.4966e-08
New model has trained and saved as: weight/EPI-DA_07180658/
Finetune done.
New model path is : weight/EPI-DA_07180658/
Makedir: cross_domain/epi/pred-07180658
Object will save in (Existing) folder/print(save_filename): cross_domain/epi/pred-07180658
Finding 2 nii.gz format files.
Reading: ASD_Xinhua_SJL_20191215_con06__E3_P1.nii.gz
(35, 90, 90)
Reading: ASD_Xinhua_SJL_20191215_tcell_0__E3_P1.1.nii.gz
(35, 90, 90)
Done.
********** Inferring CT/MRI scans: **********
********** 2 file(s) to save: **********
Saving: (35, 90, 90)
Saving: (35, 90, 90)
********** Done. **********
Note: here target domain labels are unseen, we use empty matrix as placeholder, so the ‘loss’， ‘dice_coef’, and ‘val_loss’ don’t seem to work.
## 2.3 Run baseline method
[8]:
''' zero-shot '''
zeroshot_out_folder = output_folder_base + '/' + 'pred' + '-' + 'zeroshot'
inference_pipeline(input_folder,
zeroshot_out_folder, is_mkdir=True,
weight=old_model_name, # zeroshot
# weight=new_weight,
need_rotate=need_rotate,
BN_list=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
)
Makedir: cross_domain/epi/pred-zeroshot
Object will save in (Existing) folder/print(save_filename): cross_domain/epi/pred-zeroshot
Finding 2 nii.gz format files.
Reading: ASD_Xinhua_SJL_20191215_con06__E3_P1.nii.gz
(35, 90, 90)
Reading: ASD_Xinhua_SJL_20191215_tcell_0__E3_P1.1.nii.gz
(35, 90, 90)
Done.
********** Inferring CT/MRI scans: **********
********** 2 file(s) to save: **********
Saving: (35, 90, 90)
Saving: (35, 90, 90)
********** Done. **********
## 2.4 Visualize the output
[9]:
from glob import glob
from utils.load_data import get_itk_array
from utils.visualization import load_slice, plot_segmentation
raw, zeroshot, DA = load_slice(raw_path=r'cross_domain/epi/src',
zeroshot_path=r'cross_domain/epi/pred-zeroshot',
DA_path=DA_path,
scans_num=1)
plot_segmentation(raw, zeroshot, DA, task='modality', hspace=-0.5, figsize=(16,9))
From top raw to the fifth raw: ** Raw image **, ** baseline result ** (0 label used), and ** BEN’s ** result (0 label used).
For this exemplar domain adaptation (DA) task, No label is used (zero-shot).
# 3.0 Conclusion
BEN shows the capacity to generalize well to other domains with minimal or even without additional labeled data, while the baseline approaches lead to abundant errors.
# 4.0 (Optional) Download new weight
After updating weight [1], BEN has already deployed on this new domain. You can download this weight to use next time without the abovementioned steps.
[1] In practice, we recommend using 4~6 target domain labels for domain adaptation. We also provide a semi-supervised or human-in-the-loop manner to deploy BEN.
[10]:
# import os
# from google.colab import files
# new_weight_folder = os.path.dirname(new_weight)
# print(new_weight_folder)
# os.system(f"zip -r -j download.zip {new_weight_folder}/*")
# files.download("/content/BEN/weight/weight.zip")
[10]: