Commit 7e56e69b authored by (Jay) Junqi Jiang's avatar (Jay) Junqi Jiang
Browse files

Update README.md, add references

parent 683e9e5a
**MMXAI** is a library & interface that explain Multimodal Machine Learning models.
<!-- - [Quickstart](web_app/static/docs/install.md)
- [Project Introduction](web_app/static/docs/README.md) -->
# About the project
MSc Computing Group Project - An Explainable AI (XAI) Interface for Multimodal Classification Models, Department of Computing, Imperial College London.
This project aims to explore ways to explain predicition decisions by *Multimodal* Machine Learning models and build a web app for users to interact with. Given a user model and an input examples, the software will provide interpretations for the model’s behaviour. The project started out by targeting classification problems and specifically the challenge of classifying hateful memes. The backend is packaged up as a library named MMXAI, which is powered by popular explanation methods, and aims to be model & framwork agnostic. We welcome contributors to extend it to more ML tasks / more explanation methods!
Interested? play with the [interface](www.mmxai.com) yourself!
Meet the [team](web_app/static/docs/about.md) behind this.
# Installation
## MMXAI
......@@ -19,7 +28,7 @@ pip install -r requirements.txt
## Running the web app
### Installing MMF
### Installing [MMF](https://github.com/facebookresearch/mmf)
In the web app, we use several pretrained models from MMF, who does not provice interfaces with these models. The /mmf directory contains the extended and modified files for this purpose. These are integrated in a fork of [MMF](https://github.com/junqi-jiang/mmf). We recommend install from this fork rather than install from MMF and manually replace the files.
```
......@@ -29,7 +38,7 @@ pip install --editable .
```
#### Fixing checkpoints config file
We use MMBT, late_fusion, vilbert, visual_bert models from MMF. Due to a [known issue in MMF](https://github.com/facebookresearch/mmf/issues/784), some downloaded pretrained models have curious config files which will cause error, some extra steps are needed to solve it.
We use [MMBT](https://arxiv.org/abs/1909.02950), late_fusion, [vilbert](https://arxiv.org/abs/1908.02265), [visual bert](https://arxiv.org/abs/1908.03557) models from MMF. Due to a [known issue in MMF](https://github.com/facebookresearch/mmf/issues/784), some downloaded pretrained models have curious config files which will cause error, some extra steps are needed to solve it.
when first [loading each pretrained model](https://gitlab.doc.ic.ac.uk/g207004202/explainable-multimodal-classification/-/tree/master/examples), the files are downloaded automatically to your default torch cache directory.
......@@ -46,7 +55,7 @@ This will list all the pretrained checkpoints. cd into *visual_bert.finetuned.ha
For vilbert, they are on line 217, 220, 221, 41. For visual bert, they are on line 161, 164, 165, 41.
#### Download and configure feature extractor checkpoints for vilbert and visual bert
#### Download and configure feature extractor checkpoints for vilbert and visualbert
This step is only needed for using vilbert and visual bert. Download checkpoints from [this link](https://drive.google.com/file/d/1iJ9D_sunUKiJaQWRb4iAY0LCUJu8wR3q/view?usp=sharing), unzip it, put them wherever you want.
......@@ -61,15 +70,7 @@ cd [wherever you cloned the explainable-multimodal-classification repo]
cd web_app
python3 app.py
```
# About the project
MSc Computing Group Project - An Explainable AI (XAI) Interface for Multimodal Classification Models, Department of Computing, Imperial College London.
This project aims to explore ways to explain predicition decisions by *Multimodal* Machine Learning models and build a web app for users to interact with. Given a user model and an input examples, the software will provide interpretations for the model’s behaviour. The project started out by targeting classification problems and specifically the challenge of classifying hateful memes. The backend is packaged up as library, MMXAI, which is powered by popular explanation methods, and aims to be model & framwork agnostic. We welcome contributors to extend it to more ML tasks / more explanation methods!
Interested? play with the [interface](www.mmxai.com) yourself!
Meet the [team](web_app/static/docs/about.md) behind this.
# About the repository
This repository contains the code for the following components:
......@@ -94,4 +95,4 @@ The components above are well-separated. Future developers can easily expand thi
## Other
- [Web version docs page](https://www.mmxai.com/dos/#/about)
\ No newline at end of file
- [Web version docs page](https://www.mmxai.com/dos/#/about)
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment