Progress and results
During screening, you might want to keep track of your progress and to obtain information for your stopping criteria. This section provides documentation on useful tools for these purposes.
ASReview LAB offers some insightful statistics, a progress chart, and recall chart to keep track of your screening process and help you to decide when to stop screening.
To open the statistics panel:
The summary statistics are counts of the records in your dataset.
Total records: the total number of records in your dataset.
Labeled records: the number of records that you labeled as relevant or irrelevant, including those you added as prior knowledge.
Relevant records: the number of records that you labeled as relevant, including those you added as prior knowledge.
Irrelevant records: the number of records that you labeled as irrelevant, including those you added as prior knowledge.
Irrelevant since last relevant: the number of irrelevant records you have seen since the last relevant record.
The charts on the analytics page can be useful to monitor your progress. There is a Progress and a Recall chart. The charts do not include prior knowledge and are most relevant after you have screened at least 10 records.
The progress chart plots the number of relevant records in the last 10 records that you reviewed in ASReview LAB. For example, when you reviewed 100 records, you labeled 3 relevant records between the 91st and 100th reviewed records.
The recall chart plots the number of relevant records against the number of records that you reviewed in ASReview LAB. Relevant by ASReview LAB refers to the relevant records that you labeled with the assistance of the active learning model. Random relevant refers to the relevant records that you might find if you manually reviewed the records so far without the assistance of the active learning model.
The plots can be exported as a figure:
The number of irrelevant records since last relevant will increase the longer you screen.
With Maximum as Query Strategy, you will see a decline in the number of relevant items in the plots the longer you screen. This may help to decide when to ‘stop screening <https://github.com/asreview/asreview/discussions/557>`_.
The data behind the recall plot can be used to calculate the knee-algortitm as a stopping criteria.
Mark project as finished
When you decide to stop screening, you can mark the project as finished. You can undo this at any time. To mark your project as finished:
Go to the Projects dashboard (http://localhost:5000/projects)
Hover the project you want to mark as finished and click on Options.
Click on Mark as finished.
The button to continue screening is now disabled. This can be undone by clicking again on Mark as in review.
You can export the results of your labeling to a RIS, CSV, TSV, or Excel file. A file contains all imported data including your decisions.
The following variables will be added to your dataset:
The column titled included contains the labels as provided by the user:
0= not relevant,
1= relevant and if missing it means the record is not seen during the screening process.
The column titled asreview_ranking contains an identifier to preserve the rank ordering as described above.
The file is ordered as follows:
All relevant records you have seen in the order they were shown during the screening process.
All records not seen during the screening and ordered from most to least relevant according to the last iteration of the model.
All non-relevant records are presented in the order these are shown during the screening process.
To download your results follow these steps:
Open or Create a project.
Click on Export in the menu on the left.
Select the file type for prefer: i.e. Excel, RIS, TSV, or CSV file.
Save the file to your device.
A RIS file can only be exported if a RIS file is imported.