Mapping global dynamics of benchmark creation and saturation in artificial intelligence.
Journal
Nature communications
ISSN: 2041-1723
Titre abrégé: Nat Commun
Pays: England
ID NLM: 101528555
Informations de publication
Date de publication:
10 11 2022
10 11 2022
Historique:
received:
24
03
2022
accepted:
31
10
2022
entrez:
10
11
2022
pubmed:
11
11
2022
medline:
15
11
2022
Statut:
epublish
Résumé
Benchmarks are crucial to measuring and steering progress in artificial intelligence (AI). However, recent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and increasing centralization of benchmark dataset creation. To facilitate monitoring of the health of the AI benchmarking ecosystem, we introduce methodologies for creating condensed maps of the global dynamics of benchmark creation and saturation. We curate data for 3765 benchmarks covering the entire domains of computer vision and natural language processing, and show that a large fraction of benchmarks quickly trends towards near-saturation, that many benchmarks fail to find widespread utilization, and that benchmark performance gains for different AI tasks are prone to unforeseen bursts. We analyze attributes associated with benchmark popularity, and conclude that future benchmarks should emphasize versatility, breadth and real-world utility.
Identifiants
pubmed: 36357391
doi: 10.1038/s41467-022-34591-0
pii: 10.1038/s41467-022-34591-0
pmc: PMC9649641
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
6793Informations de copyright
© 2022. The Author(s).
Références
Bioinformatics. 2005 Jun;21 Suppl 1:i47-56
pubmed: 15961493
Sci Data. 2022 Jun 17;9(1):322
pubmed: 35715466