diff --git a/README.md b/README.md index a3a18c5..05c04bc 100644 --- a/README.md +++ b/README.md @@ -1,33 +1,59 @@ -# CAT -Computerized Adaptive Testing Package, including the following models and strategies. - -* Item Response Theory (IRT) - * MaximumFisherInformation (MFI) strategy - * Kullback-Leibler Information (KLI) strategy - * Model-Agnostic Adaptive Testing (MAAT) strategy - * Bounded Ability Estimation Adaptive Testing (BECAT) strategy - * Bilevel Optimization-Based Computerized Adaptive Testing (BOBCAT) strategy - * Neural Computerized Adaptive Testing (NCAT) strategy - -* Multidimensional Item Response Theory (MIRT) - * D-Optimality (D-opt) strategy - * Multivariate Kullback-Leibler Information (MKLI) strategy - * Model-Agnostic Adaptive Testing (MAAT) strategy - * Bilevel Optimization-Based Computerized Adaptive Testing (BOBCAT) strategy - * Neural Computerized Adaptive Testing (NCAT) strategy -* Neural Cognitive Diagnosis (NCD) - * Model-Agnostic Adaptive Testing (MAAT) strategy - * Bounded Ability Estimation Adaptive Testing (BECAT) strategy - -BECAT strategy comes from paper A Bounded Ability Estimation for Computerized Adaptive Testing(https://nips.cc/virtual/2023/poster/70224) - -## Installation +# Computerized Adaptive Testing: A Python Library + +This Python library offers a streamlined solution for rapidly developing a Computerized Adaptive Testing (CAT) system. It encompasses a comprehensive suite of tools that integrate both traditional statistical methods and recent machine learning and deep learning techniques. + +[![PyPI version](https://img.shields.io/pypi/v/EduCAT.svg)](https://pypi.org/project/EduCAT/) +[![GitHub license](https://img.shields.io/github/license/bigdata-ustc/EduCAT.svg)](https://github.com/bigdata-ustc/EduCAT/blob/main/LICENSE) + +## ❗ What is CAT About? + +**Computerized Adaptive Testing (CAT)** stands as one of the earliest and most successful integrations of educational practices and computing technology. +CAT is a dynamic and interactive process between a student and a testing system. If traditional paper-and-pencil tests are "one-for-all," then CAT is "one-for-each". Each student gets a personalized test that adapts to their proficiency level and knowledge, ensuring each question accurately assesses and challenges them. CAT tailors the selection of questions to each student’s level of proficiency, thereby maximizing the accuracy of the assessment while minimizing the test length. + + +The CAT system is split into two main components that take turns: At each test step, the Cognitive Diagnosis Model (CDM), as the user model, first uses the student’s previous responses to estimate their current proficiency, based on cognitive science or psychometrics. Then, the Selection Algorithm picks the next question from the bank according to certain criteria.This two-step process repeats until a predefined stopping rule is met, and the final estimated proficiency (i.e., diagnostic report) of individual students will be fed back to themselves as the outcome of this assessment or for facilitating future learning. + +![Figure 2](./figs/cat_image_1.png) + + + +## ⚡ Contribution + +This repository implements basic functionalities of CAT. It includes the implements three types of CDM: Item Response Theory, Multidimensional Item Response Theory and Neural Cognitive Diagnosis. And each CDM has its corresponding selection algorithm: +* **[IRT](https://link.springer.com/book/10.1007/978-0-387-89976-3): Item Response Theory** + * [MFI](https://api.taylorfrancis.com/content/books/mono/download?identifierName=doi&identifierValue=10.4324/9780203056615&type=googlepdf): Maximum Fisher Information strategy + * [KLI](https://journals.sagepub.com/doi/abs/10.1177/014662169602000303): Kullback-Leibler Information strategy + * [MAAT](https://ieeexplore.ieee.org/abstract/document/9338437/): Model-Agnostic Adaptive Testing strategy + * [BECAT](https://nips.cc/virtual/2023/poster/70224): Bounded Ability Estimation Adaptive Testing strategy + * [BOBCAT](https://www.ijcai.org/proceedings/2021/0332.pdf): Bilevel Optimization-Based Computerized Adaptive Testing strategy + * [NCAT](https://ojs.aaai.org/index.php/AAAI/article/view/20399): Neural Computerized Adaptive Testing strategy +* **[MIRT](https://link.springer.com/book/10.1007/978-0-387-89976-3): Multidimensional Item Response Theory** + * [D-opt](https://www.sciencedirect.com/science/article/pii/S0169716106260188): D-Optimality strategy + * [MKLI](https://link.springer.com/article/10.1007/s11336-014-9401-5): Multivariate Kullback-Leibler Information strategy + * [MAAT](https://ieeexplore.ieee.org/abstract/document/9338437/): Model-Agnostic Adaptive Testing strategy + * [BOBCAT](https://www.ijcai.org/proceedings/2021/0332.pdf): Bilevel Optimization-Based Computerized Adaptive Testing strategy + * [NCAT](https://ojs.aaai.org/index.php/AAAI/article/view/20399): Neural Computerized Adaptive Testing strategy +* **[NCD](https://ojs.aaai.org/index.php/AAAI/article/view/6080): Neural Cognitive Diagnosis** + * [MAAT](https://ieeexplore.ieee.org/abstract/document/9338437/): Model-Agnostic Adaptive Testing strategy + * [BECAT](https://nips.cc/virtual/2023/poster/70224): Bounded Ability Estimation Adaptive Testing strategy + + + +It is worth noting that the data needs to be processed before it can be used. In the script/dataset directory, we provide the preprocessing files for the ASSISTment dataset for reference. + +## ⚡ Installation + +To make use of our work, you should do these below: Git and install by `pip` ``` pip install -e . ``` +or install from `pypi` +``` +pip install EduCAT +``` ## Quick Start @@ -45,3 +71,79 @@ tensorboard --logdir /path/to/logs to see the visualization result. + +## 📕 Machine Learning-Based Methods +### 🔍Cognitive Diagnosis Models (CDM) +Cognitive Diagnosis Model (CDM), as the user model, first uses the student's previous responses to estimate their current proficiency, based on cognitive science or psychometrics. + + + +### ✏️Selection Algorithm + Then, the \textbf{Selection Algorithm} picks the next question from the \textbf{Question Bank} according to certain criteria \cite{lord2012applications, chang1996global, bi2020quality}. Most traditional statistical criteria are informativeness metrics, e.g., selecting the question whose difficulty matches the student's current proficiency estimate, meaning the student has roughly a 50\% chance of getting it right \cite{lord2012applications}. The above process repeats until a predefined stopping rule is met, and the final estimated proficiency (i.e., diagnostic report) of individual students will be fed back to themselves as the outcome of this assessment or for facilitating future learning. + + + + + + +### 2023 - 2024 + +- BETA-CD: a Bayesian meta-learned cognitive diagnosis framework for personalized learning [Paper](https://dl.acm.org/doi/10.1609/aaai.v37i4.25629) +- Self Supervised Graph Learning for Long-Tailed Cognitive Diagnosis [Paper](https://arxiv.org/abs/2210.08169) +- Deep reinforcement learning for adaptive learning systems [Paper](https://arxiv.org/abs/2004.08410) +- A novel computerized adaptive testing framework with decoupled learning selector [Paper](https://link.springer.com/article/10.1007/s40747-023-01019-1) +- Gmocat: A graph enhanced multi-objective method for computerized adaptive testing [Paper](https://arxiv.org/abs/2310.07477) +- Towards scalable adaptive learning with graph neural net works and reinforcement learning [Paper](https://arxiv.org/abs/2305.06398) +- Towards a holistic under standing of mathematical questions with contrastive pre-training [Paper](https://arxiv.org/abs/2301.07558) +- Adaptive e-learning system based on learner portraits and knowledge graph [Paper](https://ieeexplore.ieee.org/document/10164957/) +- A bounded ability estimation for computerized adaptive testing [Paper](https://openreview.net/pdf?id=tAwjG5bM7H) +- Balancing test accuracy and security in computerized adaptive testing [Paper](https://arxiv.org/abs/2305.18312) +- Search-efficient computerized adaptive testing [Paper](https://dl.acm.org/doi/10.1145/3583780.3615049) + +### 2022-2023 + +- Hiercdf: A bayesian network-based hierarchical cognitive diagnosis framework [Paper](https://dl.acm.org/doi/10.1145/3534678.3539486) +- Deep cognitive diagnosis model for predicting students’ performance [Paper](https://www.sciencedirect.com/science/article/pii/S0167739X21003277) +- Computerized adaptive testing: A unified approach under markov decision process [Paper](https://link.springer.com/chapter/10.1007/978-3-031-10522-7_40) +- Fully adaptive framework: Neural computerized adaptive testing for online education [Paper](https://ojs.aaai.org/index.php/AAAI/article/view/20399) +- Is the naplan results delay about politics or precision? [Paper](https://blog.aare.edu.au/is-the-naplan-results-delay-about-politics-or-precision/) +- Algorithmic fairness in education [Paper](https://arxiv.org/abs/2007.05443) +- A robust computerized adaptive testing approach in educational question retrieval [Paper](https://dl.acm.org/doi/abs/10.1145/3477495.3531928) +- Self-Attention Gated Cognitive Diagnosis For Faster Adaptive Educational Assessments [Paper](https://ieeexplore.ieee.org/document/10027634/) + +### 2021-2022 + +- Item response ranking for cognitive diagnosis, in Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 [Paper](https://www.ijcai.org/proceedings/2021/241) +- Rcd: Relation map driven cognitive diagnosis for intelligent education systems [Paper](https://dl.acm.org/doi/10.1145/3404835.3462932) +- Bobcat: Bilevel optimization based computerized adaptive testing [Paper](https://www.ijcai.org/proceedings/2021/0332.pdf) +- Multi-objective optimization of item selection in computerized adaptive testing [Paper](https://dl.acm.org/doi/10.1145/3449639.3459334) +- Consistency-aware multi-modal network for hierarchical multi-label classification in online education system [Paper](https://ieeexplore.ieee.org/document/9667767/) + +### 2020-2021 + +- Neural cognitive diagnosis for intelligent education systems [Paper](https://arxiv.org/abs/1908.08733) +- Quality meets diversity: A model agnostic framework for computerized adaptive testing [Paper](https://ieeexplore.ieee.org/abstract/document/9338437/) + +### 2019-2020 + +- Dirt: Deep learning en hanced item response theory for cognitive diagnosis,” in Proceedings of the 28th ACM International Conference on Information and Knowledge [Paper](https://dl.acm.org/doi/10.1145/3357384.3358070) +- Robust computerized adaptive testing [Paper](https://link.springer.com/chapter/10.1007/978-3-030-18480-3_15) +- Reinforcement learning applied to adaptive classification testing [Paper](https://link.springer.com/chapter/10.1007/978-3-030-18480-3_17) +- Exploiting cognitive structure for adaptive learning [Paper](https://arxiv.org/abs/1905.12470) +- Question difficulty prediction for multiple choice problems in medical exams [Paper](https://dl.acm.org/doi/10.1145/3357384.3358013) +- Hierarchical multi label text classification: An attention-based recurrent network approach [Paper](https://dl.acm.org/doi/abs/10.1145/3357384.3357885) +- Quesnet: A unified representation for heterogeneous test questions [Paper](https://arxiv.org/abs/1905.10949) + +### Before 2019 + +- Recommendation system for adaptive learning [Paper](https://journals.sagepub.com/doi/full/10.1177/0146621617697959) +- Question difficulty prediction for reading problems in standard tests [Paper](https://dl.acm.org/doi/10.5555/3298239.3298437) +- Detecting biased items using catsib to increase fairness in computer adaptive tests [Paper](https://www.researchgate.net/publication/287052110_Detecting_biased_items_using_CATSIB_to_increase_fairness_in_computer_adaptive_tests) +- Evaluating knowledge structure-based adaptive testing algorithms and system development [Paper](https://www.researchgate.net/publication/266889340_Evaluating_Knowledge_Structure-based_Adaptive_Testing_Algorithms_and_System_Development) +- Applications of item response theory to practi cal testing problems [Paper](https://www.semanticscholar.org/paper/Applications-of-Item-Response-Theory-To-Practical-Lord/d0476004085419b8a44953f5cdab11442c12ffaa) +- An adaptive testing system for supporting versatile educational assessment [Paper](https://www.sciencedirect.com/science/article/pii/S0360131508000973) + diff --git a/figs/cat_image_1.png b/figs/cat_image_1.png new file mode 100644 index 0000000..0e47327 Binary files /dev/null and b/figs/cat_image_1.png differ