BEN_tutorial_Ⅲ_cross_field
Note: to replicate experiments conveniently, we did not package key functions as ‘.exe’ files here. Preprocessing is also simplified.
# 1.0 Set up
## 1.1 Verify Runtime Settings
** IMPORTANT **
In the “Runtime” menu for the notebook window, select “Change runtime type.” Ensure that the following are selected: * Runtime Type = Python 3 * Hardware Accelerator = GPU
See your Hardware Accelerator in Edit/Notebook settings in Colab Menu(On the upper left usually).
## 1.2 Set up environment and install all necessary packages
After installation, you might ** need to restart ** the runtime in order to use newly installed versions (Tensorflow 1.15).
[ ]:
!pip install SimpleITK tensorflow-gpu==1.15.4 keras==2.2.4 h5py==2.10 scikit-image==0.16.2
1.3 Download code from github.
[2]:
!git clone https://github.com/yu02019/BEN.git
Cloning into 'BEN'...
remote: Enumerating objects: 162, done.
remote: Counting objects: 100% (19/19), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 162 (delta 2), reused 10 (delta 0), pack-reused 143
Receiving objects: 100% (162/162), 6.47 MiB | 10.24 MiB/s, done.
Resolving deltas: 100% (46/46), done.
[3]:
cd BEN
/content/BEN
## 1.4 Download MRI data.
[4]:
!gdown --id 1fmyBtxNJAYFtlwVBY5IiFsmUixc2MsU9
!sh download.sh
/usr/local/lib/python3.7/dist-packages/gdown/cli.py:131: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.
category=FutureWarning,
Downloading...
From: https://drive.google.com/uc?id=1fmyBtxNJAYFtlwVBY5IiFsmUixc2MsU9
To: /content/BEN/download.sh
100% 273/273 [00:00<00:00, 505kB/s]
/usr/local/lib/python3.7/dist-packages/gdown/cli.py:131: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.
category=FutureWarning,
Downloading...
From: https://drive.google.com/uc?id=1z-VMfPvFILNcaEc9jSYzcnub7t4ZcrfJ
To: /content/BEN/cross_domain/cross_domain.zip
100% 26.8M/26.8M [00:00<00:00, 277MB/s]
Archive: cross_domain.zip
creating: 7T/
creating: 7T/src/
inflating: 7T/src/20171226_185416T1RAREs30001a001.nii.gz
inflating: 7T/src/20180321_133106T2TurboRAREs20001a001.nii.gz
inflating: 7T/src/20180510_143551T1RAREs30001a001.nii.gz
creating: epi/
creating: epi/src/
inflating: epi/src/ASD_Xinhua_SJL_20191215_con06__E3_P1.nii.gz
inflating: epi/src/ASD_Xinhua_SJL_20191215_tcell_0__E3_P1.1.nii.gz
creating: rat/
creating: rat/label/
inflating: rat/label/42d_cas_27_T2.nii.gz
creating: rat/src/
inflating: rat/src/42d_cas_27_T2.nii.gz
inflating: rat/src/42d_cas_40_T2.nii.gz
inflating: rat/src/42d_cas_44_T2.nii.gz
/usr/local/lib/python3.7/dist-packages/gdown/cli.py:131: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.
category=FutureWarning,
Downloading...
From: https://drive.google.com/uc?id=1K5qIe6o8uG7cw_Uu0H3aB0FE40YXb9Hp
To: /content/BEN/weight/weight.zip
100% 69.3M/69.3M [00:00<00:00, 211MB/s]
Archive: weight.zip
creating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/.data-00000-of-00002
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/.data-00001-of-00002
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/.index
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/checkpoint
creating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012056/
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012056/.data-00000-of-00002
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012056/.data-00001-of-00002
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012056/.index
inflating: unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012056/checkpoint
The MRI data look like:
(src: raw MRI scans; label:ground truth)
[5]:
import os
import logging
import warnings
import tensorflow as tf
warnings.filterwarnings("ignore")
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' # tf log errors only
logging.getLogger('tensorflow').setLevel(logging.ERROR)
print(tf.__version__)
1.15.4
2..0 Cross platform (11.7T -> 7T)
In this case, no label required.
## 2.1 Declare key function and path
[6]:
input_folder = r'cross_domain/7T/src'
label_folder = '' # No label required.
output_folder_base = r'cross_domain/7T' # save pred nii files.(time id based on weight)
old_model_name = r'weight/unet_fp32_all_BN_NoCenterScale_polyic_epoch15_bottle256_04012051/' # load and adjust BN base on this old model(fixed/source domain usually)
new_model_name = '7T-DA' # The prefix of new weight filename will use this string
cvs_result_base = r'cross_domain/7T' # evaluation metric in csv. (path of csv, time id)
need_rotate = True
## 2.2 Run BEN’s domain transfer.
[7]:
from utils.update_model import update_weight
# transfer BEN to new domain/dataset
new_weight = update_weight(input_folder, label_folder, need_mkdir=True, weight=old_model_name,
need_rotate=need_rotate, model_name=new_model_name,
BN_list=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1], freeze=True)
print('Finetune done.\n New model path is : ', new_weight)
# setting filename
import re
num_extract = re.findall(r'\d+', new_weight)
output_folder = output_folder_base + '/' + 'pred' + '-' + num_extract[-1] # e.g. 'pred-04061444'
DA_path = output_folder
cvs_result = cvs_result_base + '/' + new_weight.replace('weight/', '').replace('/', '') + '.csv'
new_weight = new_weight + '.hdf5'
from utils.inference import inference_pipeline
import tensorflow.keras.backend as K
K.clear_session() # release GRAM
# run inference
''' BEN (No label required) '''
inference_pipeline(input_folder,
output_folder, is_mkdir=True,
weight=new_weight,
need_rotate=need_rotate,
BN_list=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
)
Finding 3 nii.gz format files.
Reading: 20171226_185416T1RAREs30001a001.nii.gz
(9, 256, 256)
Reading: 20180321_133106T2TurboRAREs20001a001.nii.gz
(9, 256, 256)
Reading: 20180510_143551T1RAREs30001a001.nii.gz
(9, 256, 256)
Done.
Create empty label matrix!
********** Inferring CT/MRI scans: **********
Trainable lay: batch_normalization
Trainable lay: batch_normalization_1
Trainable lay: batch_normalization_2
Trainable lay: batch_normalization_3
Trainable lay: batch_normalization_4
Trainable lay: batch_normalization_5
Trainable lay: batch_normalization_6
Trainable lay: batch_normalization_7
Trainable lay: batch_normalization_8
Trainable lay: batch_normalization_9
Only finetune BN on target domain!
Using TensorFlow backend.
Train on 24 samples, validate on 3 samples
Epoch 1/30
24/24 [==============================] - 6s 231ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.2199 - val_dice_coef: 5.1076e-08
Epoch 2/30
24/24 [==============================] - 0s 4ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.3085 - val_dice_coef: 5.1647e-08
Epoch 3/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.4186 - val_dice_coef: 5.1452e-08
Epoch 4/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.5420 - val_dice_coef: 5.0847e-08
Epoch 5/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.6538 - val_dice_coef: 5.0145e-08
Epoch 6/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.7440 - val_dice_coef: 4.9609e-08
Epoch 7/30
24/24 [==============================] - 0s 4ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.8087 - val_dice_coef: 4.9313e-08
Epoch 8/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.8580 - val_dice_coef: 4.9145e-08
Epoch 9/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.8974 - val_dice_coef: 4.9058e-08
Epoch 10/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.9278 - val_dice_coef: 4.9023e-08
Epoch 11/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.9520 - val_dice_coef: 4.9016e-08
Epoch 12/30
24/24 [==============================] - 0s 4ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.9705 - val_dice_coef: 4.9028e-08
Epoch 13/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.9840 - val_dice_coef: 4.9050e-08
Epoch 14/30
24/24 [==============================] - 0s 4ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.9933 - val_dice_coef: 4.9078e-08
Epoch 15/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.9991 - val_dice_coef: 4.9108e-08
Epoch 16/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0024 - val_dice_coef: 4.9138e-08
Epoch 17/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0041 - val_dice_coef: 4.9167e-08
Epoch 18/30
24/24 [==============================] - 0s 4ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0047 - val_dice_coef: 4.9194e-08
Epoch 19/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0046 - val_dice_coef: 4.9218e-08
Epoch 20/30
24/24 [==============================] - 0s 4ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0040 - val_dice_coef: 4.9239e-08
Epoch 21/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0033 - val_dice_coef: 4.9257e-08
Epoch 22/30
24/24 [==============================] - 0s 4ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0025 - val_dice_coef: 4.9273e-08
Epoch 23/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0018 - val_dice_coef: 4.9286e-08
Epoch 24/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0011 - val_dice_coef: 4.9297e-08
Epoch 25/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0005 - val_dice_coef: 4.9306e-08
Epoch 26/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 4.0000 - val_dice_coef: 4.9313e-08
Epoch 27/30
24/24 [==============================] - 0s 3ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.9995 - val_dice_coef: 4.9319e-08
Epoch 28/30
24/24 [==============================] - 0s 4ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.9992 - val_dice_coef: 4.9324e-08
Epoch 29/30
24/24 [==============================] - 0s 4ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.9989 - val_dice_coef: 4.9328e-08
Epoch 30/30
24/24 [==============================] - 0s 4ms/sample - loss: 4.9004 - dice_coef: 5.4996e-09 - val_loss: 3.9986 - val_dice_coef: 4.9331e-08
New model has trained and saved as: weight/7T-DA_07180711/
Finetune done.
New model path is : weight/7T-DA_07180711/
Makedir: cross_domain/7T/pred-07180711
Object will save in (Existing) folder/print(save_filename): cross_domain/7T/pred-07180711
Finding 3 nii.gz format files.
Reading: 20171226_185416T1RAREs30001a001.nii.gz
(9, 256, 256)
Reading: 20180321_133106T2TurboRAREs20001a001.nii.gz
(9, 256, 256)
Reading: 20180510_143551T1RAREs30001a001.nii.gz
(9, 256, 256)
Done.
********** Inferring CT/MRI scans: **********
********** 3 file(s) to save: **********
Saving: (9, 256, 256)
Saving: (9, 256, 256)
Saving: (9, 256, 256)
********** Done. **********
Note: here target domain labels are unseen, we use empty matrix as placeholder, so the ‘loss’， ‘dice_coef’, and ‘val_loss’ don’t seem to work.
## 2.3 Run baseline method
[8]:
''' zero-shot '''
zeroshot_out_folder = output_folder_base + '/' + 'pred' + '-' + 'zeroshot'
inference_pipeline(input_folder,
zeroshot_out_folder, is_mkdir=True,
weight=old_model_name, # zeroshot
need_rotate=need_rotate,
BN_list=[1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
)
Makedir: cross_domain/7T/pred-zeroshot
Object will save in (Existing) folder/print(save_filename): cross_domain/7T/pred-zeroshot
Finding 3 nii.gz format files.
Reading: 20171226_185416T1RAREs30001a001.nii.gz
(9, 256, 256)
Reading: 20180321_133106T2TurboRAREs20001a001.nii.gz
(9, 256, 256)
Reading: 20180510_143551T1RAREs30001a001.nii.gz
(9, 256, 256)
Done.
********** Inferring CT/MRI scans: **********
********** 3 file(s) to save: **********
Saving: (9, 256, 256)
Saving: (9, 256, 256)
Saving: (9, 256, 256)
********** Done. **********
## 2.4 Visualize the output
[9]:
''' cross field '''
from glob import glob
from utils.load_data import get_itk_array
from utils.visualization import load_slice, plot_segmentation
raw, zeroshot, DA = load_slice(raw_path=r'cross_domain/7T/src',
zeroshot_path=r'cross_domain/7T/pred-zeroshot',
DA_path=DA_path,
scans_num=1)
plot_segmentation(raw, zeroshot, DA, task='field strength', figsize=(16,9))
From top raw to the fifth raw: ** Raw image **, ** baseline result ** (0 label used), and ** BEN’s ** result (0 label used).
For this exemplar domain adaptation (DA) task, No label is used (zero-shot).
# 3.0 Conclusion
BEN shows the capacity to generalize well to other domains with minimal or even without additional labeled data, while the baseline approaches lead to abundant errors.
# 4.0 (Optional) Download new weight
After updating weight [1], BEN has already deployed on this new domain. You can download this weight to use next time without the abovementioned steps.
[1] In practice, we recommend using 4~6 target domain labels for domain adaptation. We also provide a semi-supervised or human-in-the-loop manner to deploy BEN.
[10]:
# import os
# from google.colab import files
# new_weight_folder = os.path.dirname(new_weight)
# print(new_weight_folder)
# os.system(f"zip -r -j download.zip {new_weight_folder}/*")
# files.download("/content/BEN/weight/weight.zip")
[ ]: