- In progress
- Research
- AI/ML
- Solo
- 2025 - present
Multi-teacher distillation for chest X-rays
Master's thesis: three different teachers trained on 112,120 chest X-rays, distilled into one lightweight student.
How it works
- 01NIH ChestX-ray14: 112,120 images, 14 labels
- 02Teachers: DenseNet-121, ResNet-50, RAD-DINO ViT
- 03Uncertainty-based teacher weighting
- 04Student: EfficientNet-B0
Best teacher mAUCRAD-DINO ViT on 112,120 NIH chest X-rays; CheXNet reports 0.841.
Best teacher mAUC
RAD-DINO ViT on 112,120 NIH chest X-rays; CheXNet reports 0.841.
ResNet-50 mAUC after tuningUp from 0.699 with hyperparameter search.
ResNet-50 mAUC after tuning
Up from 0.699 with hyperparameter search.
Three teachers, one student
Pick a teacher to see its mAUC on the full NIH test set.
The student (orange) is still training; no student result is claimed yet.
The problem
Chest X-ray labels are long-tailed: some findings have tens of thousands of examples, others a few hundred. One large model is hard to deploy, and a small one forgets the rare classes.
My part
Trained and tuned all three teachers and the distillation pipeline myself, as my thesis.
What did not work
- ResNet-50 started at 0.699 mAUC; hyperparameter search lifted it to 0.804.
Limitations
- Distillation into the student is still running; no student result is claimed yet.
