arxiv:2409.03643

CDM: A Reliable Metric for Fair and Accurate Formula Recognition Evaluation

Published on Sep 5

· Submitted by

wanderkid on Sep 6

Upvote

Authors:

Bin Wang ,

Linke Ouyang ,

Renqiu Xia ,

Bo Zhang ,

Conghui He

Abstract

Formula recognition presents significant challenges due to the complicated structure and varied notation of mathematical expressions. Despite continuous advancements in formula recognition models, the evaluation metrics employed by these models, such as BLEU and Edit Distance, still exhibit notable limitations. They overlook the fact that the same formula has diverse representations and is highly sensitive to the distribution of training data, thereby causing the unfairness in formula recognition evaluation. To this end, we propose a Character Detection Matching (CDM) metric, ensuring the evaluation objectivity by designing a image-level rather than LaTex-level metric score. Specifically, CDM renders both the model-predicted LaTeX and the ground-truth LaTeX formulas into image-formatted formulas, then employs visual feature extraction and localization techniques for precise character-level matching, incorporating spatial position information. Such a spatially-aware and character-matching method offers a more accurate and equitable evaluation compared with previous BLEU and Edit Distance metrics that rely solely on text-based character matching. Experimentally, we evaluated various formula recognition models using CDM, BLEU, and ExpRate metrics. Their results demonstrate that the CDM aligns more closely with human evaluation standards and provides a fairer comparison across different models by eliminating discrepancies caused by diverse formula representations.

View arXiv page View PDF Add to collection

Community

wanderkid

Paper author Paper submitter Sep 6

•

edited Sep 6

BLEU and Edit Distance are commonly used metrics in fields such as machine translation and text recognition. While these metrics have also been applied to formula recognition, they fall short due to the non-unique representation of LaTeX formulas, leading to inaccurate assessments and unfair comparisons. The proposed Character Detection Matching (CDM) metric addresses these limitations by employing an image-level character detection and matching approach. This method ensures accurate and fair evaluation of formula recognition capabilities, which is crucial for advancing the field.

Project page here: https://github.com/opendatalab/UniMERNet/tree/main/cdm