|
|
--- |
|
|
datasets: |
|
|
- QCRI/CrisisMMD |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- accuracy |
|
|
- f1 |
|
|
- recall |
|
|
- precision |
|
|
base_model: |
|
|
- google-bert/bert-base-uncased |
|
|
- microsoft/resnet-50 |
|
|
--- |
|
|
Source: CrisisMMD dataset (Alam et al., 2017) |
|
|
|
|
|
✅Original Labels (8 classes from annotations): |
|
|
|
|
|
Infrastructure and utility damage |
|
|
|
|
|
Vehicle damage |
|
|
|
|
|
Rescue, volunteering, or donation efforts |
|
|
|
|
|
Affected individuals |
|
|
|
|
|
Injured or dead people |
|
|
|
|
|
Missing or found people |
|
|
|
|
|
Other relevant information |
|
|
|
|
|
Not humanitarian |
|
|
|
|
|
✅Label Preprocessing (Class Merging): |
|
|
|
|
|
Vehicle damage merged into Infrastructure and utility damage |
|
|
|
|
|
Missing or found people merged into Affected individuals |
|
|
|
|
|
Not humanitarian retained as a separate class |
|
|
|
|
|
Removed very low-frequency categories (e.g., "Missing or found people" as a separate class) |
|
|
|
|
|
✅Final Label Set (5 classes total): |
|
|
|
|
|
Infrastructure and utility damage |
|
|
|
|
|
Rescue, volunteering, or donation efforts |
|
|
|
|
|
Affected individuals |
|
|
|
|
|
Injured or dead people |
|
|
|
|
|
Not humanitarian |
|
|
|
|
|
✅Multimodal Consistency: |
|
|
|
|
|
Selected only those posts where text and image annotations matched |
|
|
|
|
|
Resulted in a total of 8,219 consistent samples: |
|
|
|
|
|
Train set: 6,574 posts |
|
|
|
|
|
Test set: 1,644 posts |
|
|
|
|
|
✅ Preprocessing Done |
|
|
Text: |
|
|
|
|
|
Tokenized using BERT tokenizer (bert-base-uncased) |
|
|
|
|
|
Extracted input_ids and attention_mask |
|
|
|
|
|
Image: |
|
|
|
|
|
Processed using ResNet-50 |
|
|
|
|
|
Extracted 2048-dimensional image features |
|
|
|
|
|
The preprocessed data was saved in PyTorch .pt format: |
|
|
|
|
|
train_human.pt and test_human.pt |
|
|
|
|
|
Each contains: input_ids, attention_mask, image_vector, and label |
|
|
|
|
|
✅ Model Architecture |
|
|
A custom multimodal classifier that combines BERT and ResNet-50 outputs: |
|
|
|
|
|
Component Details |
|
|
Text Encoder BERT base (bert-base-uncased) – outputs pooler_output (768-d) |
|
|
Image Encoder Pre-extracted ResNet-50 image features (2048-d) |
|
|
Fusion Concatenation → FC layers → Softmax over 5 classes |
|
|
Classifier Fully connected layers with BatchNorm, ReLU, Dropout |
|
|
|
|
|
✅ Training Setup |
|
|
Loss Function: CrossEntropyLoss |
|
|
|
|
|
Optimizer: AdamW |
|
|
|
|
|
Scheduler: StepLR (γ = 0.9) |
|
|
|
|
|
Epochs Tried: 1, 3, 5, 8, 10 |
|
|
|
|
|
Batch Size: 16 |
|
|
|
|
|
Runtime: ~2 minutes 20 seconds per epoch on Google Colab (T4 GPU) |
|
|
|
|
|
✅ Evaluation Metrics |
|
|
Accuracy |
|
|
|
|
|
Precision |
|
|
|
|
|
Recall |
|
|
|
|
|
F1 Score |
|
|
|
|
|
✅ Metrics(epoch 3 with highest accuracy) |
|
|
|
|
|
✅ Test Accuracy : 0.8820 |
|
|
✅ Precision : 0.6854 |
|
|
✅ Recall : 0.7176 |
|
|
✅ F1 Score : 0.7005 |
|
|
|
|
|
The new dataset created: https://huggingface.co/datasets/Henishma/crisisMMD_cleaned_task2 |
|
|
|
|
|
|
