EMC2 Challenge 2020 FAQs

Latest Update: 06/16/2020

General FAQ

Q : Who can participate the EMC2 challenge?

A : Anyone is welcomed

Q : What are the metrics to rank the submissions ?

A : Accuracy and Model Size

Q : When can I submit?

A :June 22, 2020

Q : What the results will be announced?

A : We will announce the winners in the same time as NeurIPS 2020.

Q : What resources are available to get familiar with model compression?

A1 : Review article: https://arxiv.org/abs/2006.03669

A2 : List of papers with code: https://paperswithcode.com/task/model-compression

A3 : TensorFlow model optimization toolkit: https://github.com/tensorflow/model-optimization

A4 : AI Model Efficiency Toolkit (AIMET): https://github.com/quic/aimet

A5 : Neural network distiller: https://github.com/NervanaSystems/distiller

Q : Is Transformer model allowed in the challenge?

A : All models are allowed in this challenge.

Q : How is the model size evaluated?

A : We only look at the physical bytes of the model file. We don't consider the runtime memory footprint.

Q : Which NLP tasks are evaluated by the model? Is per task model allowed?

A : We will evaluate the NLP model on all GLUE tasks (https://gluebenchmark.com/leaderboard) and will weight all the tasks equally. Any model or strategies are welcome!