Merge branch 'master' of gitlab.doc.ic.ac.uk:aa19418/nlp-cw

README.md

@@ -9,29 +9,36 @@
     - back translation (ella)
     - feature space synonym replacemnet (emily)
 
-# hyperparameter
-simple transformers.ai
-- learning rate
-- train_batch_size
-- num_train_epochs
+1. hyperparameter tuning on  roberta-base
+azhara
+- learning rate [0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01] 
+- optimizer on [AdamW, Adafactor]
+
+ella
 - early stopping (only req for longer epochs)
-- optimizer on (adam w and ada factor)
-- scheduler
-- logging_steps
-- model (each try one model)
+- num_train_epochs [1, 5, 10, 15, 20]
+
+emily
+- train_batch_size [8, 16, 32, 64, 128]
+- scheduler ["linear_schedule_with_warmup", "polynomial_decay_schedule_with_warmup", "constant_schedule_with_warmup"]
 
+2. cased or uncased
 
-# for augmentation
+3. augmentation parameter tuning
 - percentage of word embeddings replaced in BERT (em)
     - how much percentage of all sentences
 
 - synonym (azhara)
     - percentage of words replacing
 
-- back translation ()
+- back translation (ella)
     - which languages, and amount of languages
 
 
+4. other
+- model ["facebook/bart-large-cnn", "distilroberta-base", "bert-base-cased"]
+- do a larger/smaller version for each model above
+
 - evaluate