initial commit:

README.md

@@ -0,0 +1,60 @@
+# Mutlimodality for skin lesions classification
+
+Many people worldwide suffer from skin diseases. For diagnosis, physicians often combine multiple information sources. These include, for instance, clinical images, microscopic images and meta-data such as the age and gender of the patient. Deep learning algorithms can support the classification of skin lesions by fusing all the information together and evaluating it. Several such algorithms are already being developed. However, to apply these learning algorithms in the clinic, they need to be further improved to achieve higher diagnostic accuracy.
+
+## Dataset
+
+Download the [ISIC 2020 dataset](https://www.kaggle.com/nroman/melanoma-external-malignant-256).
+In the directory you will find:
+- metadata as `train.csv` and `test.csv`,
+- images for train and test subsets.
+
+## Training multimodal EfficientNet
+
+In its most basic form, training new networks boils down to:
+
+```.bash
+python train.py --save-name efficientnetb2_256_20ep --data-dir ./melanoma_external_256/ --image-size 256 \
+                --n-epochs 20 --enet-type efficientnet-b2 --CUDA_VISIBLE_DEVICES 0
+python train.py --save-name efficientnetb2_256_20ep_meta --data-dir ./melanoma_external_256/ --image-size 256 \
+                --n-epochs 20 --enet-type efficientnet-b2 --CUDA_VISIBLE_DEVICES 0 --use-meta
+```
+
+The first command is uses only images during training; for the second one additional addition of avalilable metadata is done.
+
+## Training multilabel classifier
+
+We created a model with multiple binary heads to distinguish between different type of biases, such as ruler and black frame.
+To use the model check `multi_classification.py` script.
+
+```.bash
+python multi_classification.py --img_path ./melanoma_external_256/train/train --ann_path gans_biases.csv \
+                               --mode train --model_path multiclasificator_efficientnet-b2_GAN.pth
+
+python multi_classification.py --img_path ./melanoma_external_256/train/train --ann_path gans_biases.csv \
+                               --mode val --model_path multiclasificator_efficientnet-b2_GAN.pth
+
+python multi_classification.py --img_path ./melanoma_external_256/test/test --mode test \
+                               --model_path multiclasificator_efficientnet-b2_GAN.pth --save_path annotations.csv
+```
+
+We can distinguish between 3 modes:
+- train: we need here provided annotations of biases for each image,
+- val: we need here provided annotations of biases for each image and trained model,
+- test: we need trained model to create new annotations for unseen images.
+
+## Creditentials
+
+This project based on code produced by [1st place on liderboard for Kaggle ISIC 2020 competition](https://www.kaggle.com/c/siim-isic-melanoma-classification/leaderboard).
+
+More details can be found here:
+
+https://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution
+
+https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175412
+
+http://arxiv.org/abs/2010.05351
+
+## Acknowledgements
+
+The project was developed during the first rotation of the [Eye for AI Program](https://www.ai.se/en/eyeforai) at the AI Competence Center of [Sahlgrenska University Hospital](https://www.sahlgrenska.se/en/). Eye for AI initiative is a global program focused on bringing more international talents into the Swedish AI landscape.

dataset.py

@@ -0,0 +1,150 @@
+import os
+import cv2
+import numpy as np
+import pandas as pd
+import albumentations
+import torch
+from torch.utils.data import Dataset
+
+from tqdm import tqdm
+
+
+class MelanomaDataset(Dataset):
+    def __init__(self, csv, mode, meta_features, transform=None):
+
+        self.csv = csv.reset_index(drop=True)
+        self.mode = mode
+        self.use_meta = meta_features is not None
+        self.meta_features = meta_features
+        self.transform = transform
+
+    def __len__(self):
+        return self.csv.shape[0]
+
+    def __getitem__(self, index):
+
+        row = self.csv.iloc[index]
+
+        image = cv2.imread(row.filepath)
+        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
+
+        if self.transform is not None:
+            res = self.transform(image=image)
+            image = res['image'].astype(np.float32)
+        else:
+            image = image.astype(np.float32)
+
+        image = image.transpose(2, 0, 1)
+
+        if self.use_meta:
+            data = (torch.tensor(image).float(), torch.tensor(self.csv.iloc[index][self.meta_features]).float())
+        else:
+            data = torch.tensor(image).float()
+
+        if self.mode == 'test':
+            return data
+        else:
+            return data, torch.tensor(self.csv.iloc[index].target).long()
+
+
+def get_transforms(image_size):
+
+    transforms_train = albumentations.Compose([
+        albumentations.Transpose(p=0.5),
+        albumentations.VerticalFlip(p=0.5),
+        albumentations.HorizontalFlip(p=0.5),
+        albumentations.RandomBrightness(limit=0.2, p=0.75),
+        albumentations.RandomContrast(limit=0.2, p=0.75),
+        albumentations.OneOf([
+            albumentations.MotionBlur(blur_limit=5),
+            albumentations.MedianBlur(blur_limit=5),
+            albumentations.GaussianBlur(blur_limit=5),
+            albumentations.GaussNoise(var_limit=(5.0, 30.0)),
+        ], p=0.7),
+
+        albumentations.OneOf([
+            albumentations.OpticalDistortion(distort_limit=1.0),
+            albumentations.GridDistortion(num_steps=5, distort_limit=1.),
+            albumentations.ElasticTransform(alpha=3),
+        ], p=0.7),
+
+        albumentations.CLAHE(clip_limit=4.0, p=0.7),
+        albumentations.HueSaturationValue(hue_shift_limit=10, sat_shift_limit=20, val_shift_limit=10, p=0.5),
+        albumentations.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),
+        albumentations.Resize(image_size, image_size),
+        albumentations.Cutout(max_h_size=int(image_size * 0.375), max_w_size=int(image_size * 0.375), num_holes=1, p=0.7),
+        albumentations.Normalize()
+    ])
+
+    transforms_val = albumentations.Compose([
+        albumentations.Resize(image_size, image_size),
+        albumentations.Normalize()
+    ])
+
+    return transforms_train, transforms_val
+
+
+def get_meta_data(df_train, df_test):
+    df_train['sex'].fillna(df_train['sex'].mode()[0], inplace=True)
+    df_train['age_approx'].fillna(df_train['age_approx'].median(), inplace=True)
+    df_train['anatom_site_general_challenge'].fillna('unknown', inplace=True)
+    df_test['anatom_site_general_challenge'].fillna('unknown', inplace=True)
+    df_test['sex'].fillna(df_test['sex'].mode()[0], inplace=True)
+    df_test['age_approx'].fillna(df_test['age_approx'].median(), inplace=True)
+
+    # One-hot encoding of anatom_site_general_challenge feature
+    concat = pd.concat([df_train['anatom_site_general_challenge'], df_test['anatom_site_general_challenge']], ignore_index=True)
+    dummies = pd.get_dummies(concat, dummy_na=True, dtype=np.uint8, prefix='site')
+    df_train = pd.concat([df_train, dummies.iloc[:df_train.shape[0]]], axis=1)
+    df_test = pd.concat([df_test, dummies.iloc[df_train.shape[0]:].reset_index(drop=True)], axis=1)
+    # Sex features
+    df_train['sex'] = df_train['sex'].map({'male': 1, 'female': 0})
+    df_test['sex'] = df_test['sex'].map({'male': 1, 'female': 0})
+    # Age features
+    df_train['age_approx'] /= 90
+    df_test['age_approx'] /= 90
+    # patient id
+    df_train['patient_id'] = df_train['patient_id'].fillna(0)
+    # n_image per user
+    df_train['n_images'] = df_train.patient_id.map(df_train.groupby(['patient_id']).image_name.count())
+    df_test['n_images'] = df_test.patient_id.map(df_test.groupby(['patient_id']).image_name.count())
+    df_train.loc[df_train['patient_id'] == -1, 'n_images'] = 1
+    df_train['n_images'] = np.log1p(df_train['n_images'].values)
+    df_test['n_images'] = np.log1p(df_test['n_images'].values)
+    # image size
+    train_images = df_train['filepath'].values
+    train_sizes = np.zeros(train_images.shape[0])
+    for i, img_path in enumerate(tqdm(train_images)):
+        train_sizes[i] = os.path.getsize(img_path)
+    df_train['image_size'] = np.log(train_sizes)
+    test_images = df_test['filepath'].values
+    test_sizes = np.zeros(test_images.shape[0])
+    for i, img_path in enumerate(tqdm(test_images)):
+        test_sizes[i] = os.path.getsize(img_path)
+    df_test['image_size'] = np.log(test_sizes)
+
+    meta_features = ['sex', 'age_approx', 'n_images', 'image_size'] + [col for col in df_train.columns if col.startswith('site_')]
+    n_meta_features = len(meta_features)
+    return df_train, df_test, meta_features, n_meta_features
+
+
+def get_df(data_dir, use_meta):
+
+    df_train = pd.read_csv(os.path.join(data_dir, 'train.csv'))
+    df_train['filepath'] = df_train['image_name'].apply(lambda x: os.path.join(data_dir, f'train/train', f'{x}.jpg'))
+
+    df_train['is_ext'] = 0
+
+    # test data
+    df_test = pd.read_csv(os.path.join(data_dir, 'test.csv'))
+    df_test['filepath'] = df_test['image_name'].apply(lambda x: os.path.join(data_dir, f'test/test', f'{x}.jpg'))
+
+    if use_meta:
+        df_train, df_test, meta_features, n_meta_features = get_meta_data(df_train, df_test)
+    else:
+        meta_features = None
+        n_meta_features = 0
+
+    # class mapping
+    mel_idx = 1
+    return df_train, df_test, meta_features, n_meta_features, mel_idx

models.py

@@ -0,0 +1,65 @@
+import torch
+import torch.nn as nn
+from efficientnet_pytorch import EfficientNet
+
+
+sigmoid = nn.Sigmoid()
+
+
+class Swish(torch.autograd.Function):
+    @staticmethod
+    def forward(ctx, i):
+        result = i * sigmoid(i)
+        ctx.save_for_backward(i)
+        return result
+    @staticmethod
+    def backward(ctx, grad_output):
+        i = ctx.saved_variables[0]
+        sigmoid_i = sigmoid(i)
+        return grad_output * (sigmoid_i * (1 + i * (1 - sigmoid_i)))
+
+
+class Swish_Module(nn.Module):
+    def forward(self, x):
+        return Swish.apply(x)
+
+
+class Effnet_Melanoma(nn.Module):
+    def __init__(self, enet_type, out_dim, n_meta_features=0, n_meta_dim=[512, 128]):
+        super(Effnet_Melanoma, self).__init__()
+        self.n_meta_features = n_meta_features
+        self.enet = EfficientNet.from_pretrained(enet_type)
+        self.dropouts = nn.ModuleList([
+            nn.Dropout(0.5) for _ in range(5)
+        ])
+        in_ch = self.enet._fc.in_features
+        if n_meta_features > 0:
+            self.meta = nn.Sequential(
+                nn.Linear(n_meta_features, n_meta_dim[0]),
+                nn.BatchNorm1d(n_meta_dim[0]),
+                Swish_Module(),
+                nn.Dropout(p=0.3),
+                nn.Linear(n_meta_dim[0], n_meta_dim[1]),
+                nn.BatchNorm1d(n_meta_dim[1]),
+                Swish_Module(),
+            )
+            in_ch += n_meta_dim[1]
+        self.myfc = nn.Linear(in_ch, out_dim)
+        self.enet._fc = nn.Identity()
+
+    def extract(self, x):
+        x = self.enet(x)
+        return x
+
+    def forward(self, x, x_meta=None):
+        x = self.extract(x).squeeze(-1).squeeze(-1)
+        if self.n_meta_features > 0:
+            x_meta = self.meta(x_meta)
+            x = torch.cat((x, x_meta), dim=1)
+        for i, dropout in enumerate(self.dropouts):
+            if i == 0:
+                out = self.myfc(dropout(x))
+            else:
+                out += self.myfc(dropout(x))
+        out /= len(self.dropouts)
+        return out