AesCode Infographics Content: What It Measures, How It Runs, and Why Direct Comparison Is Blocked

Introduction: The Role of the Content Metric in AesCode

Generating infographics as code presents a dual challenge: ensuring data accuracy while achieving acceptable aesthetic quality. The AesCode framework, developed by Microsoft and collaborators, addresses this problem by explicitly separating content verification from visual evaluation. The 'Content' metric is a critical component of the 'Visual' score in the 300-sample infographic benchmark.

Unlike the 'Chart' and 'Bound' metrics, which are integrated into the 'Rule' score based on deterministic verifiers, 'Content' belongs to the perceptual evaluation domain. Its objective is not to verify if a number appears in the code, but whether the element required by the prompt is present and complete in the final visual representation. This distinction is fundamental to understanding why AesCode results are not directly transferable to other code or image generation benchmarks.

Separating these dimensions matters because a rendered infographic can satisfy one kind of requirement while failing another. A requested visual element may be present and recognizable even if its displayed figures are wrong; conversely, a value may be encoded correctly without the required element being clearly represented in the final image. Content is intended to assess the former question of semantic presence and completeness, while the other metrics address distinct aspects of the result. The distinction makes the score more interpretable, but it also means Content should not be treated as a standalone verdict on overall correctness.

The benchmark contains 300 infographic samples, and its reported scores are averaged over three generations per prompt. This gives the evaluation a defined scope: Content reflects checklist judgments across the benchmark’s prompts and generated outputs, rather than a single image or a general measure of infographic quality. The aggregate can help compare performance within the AesCode evaluation setup, but it does not by itself show which particular requirement was missed, whether errors were concentrated in a subset of prompts, or how performance would transfer to a different benchmark. Those questions require access to more detailed evaluation results and criteria.

Suite Structure: Metric Integration

The AesCode benchmark structures evaluation into two main families: 'Rule' and 'Visual'. The 'Rule' score is the average of the deterministic metrics Text, Bound, and Chart. These are calculated using verifiers that inspect the HTML/CSS code, the DOM, and the structured data of tables and ECharts charts.

The 'Visual' score, in contrast, is the average of three dimensions evaluated by a Vision-Language Model (VLM): Content, Layout, and Style. The Content metric focuses on whether requested elements are present and complete. Layout evaluates organization, relationships, and collisions. Style judges hierarchy, typography, and finish. The 'Overall' score is the final average between Rule and Visual. This architecture allows identifying whether a model is precise but ugly, or aesthetic but inaccurate.

The two families provide complementary evidence rather than interchangeable measurements. Rule uses deterministic checks on properties that can be inspected in code, the document structure, or structured chart and table data. Visual instead assesses the rendered result through a VLM checklist. This division reflects a practical distinction: some requirements can be tested by checking an explicit value or geometric constraint, whereas whether a requested component is visibly present and complete calls for interpretation of the image. A score in one family therefore cannot be assumed to establish success in the other.

The averaging structure also affects how to read the final results. Overall combines Rule and Visual, while each family itself combines several metrics. A strong Overall score can conceal a weaker component, and a lower family score can arise from different underlying dimensions. For example, a high Rule score does not establish that every requested visual element is present, just as a high Visual score does not establish that table or chart values are numerically correct. Looking at Content alongside Layout, Style, Text, Bound, and Chart preserves distinctions that would disappear if only the aggregate were reported.

Because the metrics differ in what they inspect and how they are evaluated, their scores should be interpreted as parts of the benchmark’s defined measurement system. The deterministic checks and the VLM-based judgments answer different questions; averaging them is a way to summarize those results, not evidence that their methods are equivalent. The family breakdown is therefore useful when diagnosing whether a result is limited by exact text or data checks, by the presence of requested elements, or by organization and visual finish.

Methodology and Execution of the Content Metric

The execution of the Content metric does not depend on fixed syntax rules, but on an evaluation process by a VLM. From the prompt requirements, a list of sample-specific yes/no questions is generated. These questions are linked to the nodes and relationships of the target 'design graph'.

The VLM answers these questions by observing only the rendering of the generated code. The Content score is the pass rate over these questions. It is crucial to highlight that the exact weighting formula between questions and the complete rubric generation protocol are not detailed in public sources in a way that allows independent replication without the full source code of the evaluation harness.

The sample-specific checklist gives Content a more targeted role than a general visual-similarity judgment. Questions are derived from the requirements of the prompt and connected to nodes and relationships in the target design graph. In this setup, the evaluator is asked to judge whether required components appear in the rendered output, rather than whether the image resembles a reference pixel for pixel. The graph connection supplies structure for formulating the questions, but the publicly described process does not establish the exact weighting assigned to each question or disclose a complete, independently reproducible rubric-generation procedure.

The evaluation chain contains several decisions that can affect the resulting score: how prompt requirements are extracted, how they are represented in the graph, how yes/no questions are formulated, and how the VLM interprets the rendered image. The described pass rate summarizes the answers to those questions, but it should not be read as a direct measurement of every aspect of semantic quality. In particular, the score depends on the checklist that was generated for the sample and on the VLM’s judgments against it. Without the full evaluation harness and rubric details, an external evaluator cannot assume that an independently written checklist would yield the same result.

The rendering step is important to the metric’s meaning. The VLM assesses the visual output of the generated code, not merely the source text or the presence of a data value in the implementation. That makes Content relevant to whether requirements survive the transition from prompt and code into the final representation. At the same time, this visual focus does not make Content a substitute for the deterministic checks: the rendered appearance can be judged for element presence while numerical accuracy remains a separate question for Chart.

Limitations and Comparability with Other Benchmarks

One of the main limitations of the AesCode Content metric is its dependence on a specific VLM and a dynamically generated rubric. This prevents direct comparison with benchmarks that use deterministic metrics or other VLMs with different judgment criteria. Consistency with human judgment is high according to the authors, but the lack of access to the complete dataset and evaluation code limits independent verification.

Furthermore, the Content metric in AesCode focuses on the presence of elements, not on the accuracy of numerical data, which is the responsibility of the Chart metric. This means a model can obtain a high Content score but fail in data accuracy, something that would not be captured in a purely visual evaluation. For a rigorous evaluation, it is necessary to consider the Rule score alongside the Visual score.

These limitations affect both interpretation and reproducibility. A VLM-based checklist can make judgments that are difficult to express as fixed syntax rules, but its results depend on the evaluator and on how the rubric is produced. If another benchmark uses different questions, a different VLM, or deterministic tests instead, equal numerical scores would not necessarily represent equal performance. The score is most informative when read with its evaluation procedure in view, rather than treated as a universal scale of semantic fidelity.

The authors report high consistency with human judgment, but the available information does not provide a basis for independent verification of that claim. In the absence of the complete dataset and evaluation code, readers cannot fully inspect the sample-level judgments, reproduce the rubric generation, or test how consistently the same rendered outputs would be scored under the published procedure. This does not invalidate the reported results; it means that their evidential scope should remain limited to the evaluation setup that is described.

A Content result also needs to be separated from the question of data correctness. A checklist may indicate that the required visual components are present and complete while numerical values in a chart or table remain inaccurate. The reverse can also occur: data checks may pass even when a required component is absent or incomplete in the rendering. Reading Rule together with Visual, and examining the underlying metrics where available, helps avoid interpreting success on one dimension as proof of success on another.

For comparisons across evaluations, normalization would need to address more than the displayed score scale. The prompts, requirements, checklist formulation, evaluator, and meaning of a passing answer all influence what is being measured. Without comparable criteria, an apparent numerical match between Content and a different benchmark’s similarity or accuracy score could be misleading. The caution is therefore methodological: compare the construct and evaluation procedure first, and treat raw scores as specific to their benchmark unless equivalence has been established.

Conclusion and Recommendations for ML Teams

The AesCode Content metric is a valuable tool for evaluating the semantic fidelity of infographics generated as code. Its integration into a multi-metric evaluation system allows for a more complete view of model quality. However, its use must be contextualized within the complete AesCode suite, including the Rule metrics.

For teams wishing to use this metric, it is recommended to access the official GitHub repository to obtain the evaluation code and sample data. The implementation requires an environment with Playwright, PaddleOCR, and a compatible VLM. The lack of detailed documentation on rubric generation is a gap that must be considered when interpreting results.

For an ML team, the metric is most useful as one diagnostic signal within the full suite. It can help assess whether a generated infographic includes the elements called for by its prompt, while the Rule metrics provide separate checks on text, geometry, and chart or table data. Layout and Style add further visual judgments. This division supports more precise analysis of model behavior than a single aggregate score, provided the team retains the distinction between what each metric actually tests and what remains outside its scope.

The repository and sample data are relevant for teams seeking to inspect or run the evaluation, but access to those materials does not remove the stated documentation gap around rubric generation. A faithful replication depends on the evaluation harness and its implementation details, not only on the names of the tools involved. Playwright, PaddleOCR, and a compatible VLM are part of the described environment, but their availability alone does not specify every decision required to reproduce the reported checklist judgments. Results from a local implementation should therefore be accompanied by a clear account of the version and evaluation procedure used.

The example scores illustrate why the separate dimensions matter without establishing a general rule for interpreting every score. A Content result of 86.41 is presented as indicating that most required elements were included, while a Rule result of 90.07 is associated with precision in data and text; the example does not specify that these figures summarize multiple generations or establish that the model will perform similarly on other prompts. Likewise, the Style value of 53.21 is used to illustrate how a design could be functional yet not aesthetically polished. These are qualitative readings of the stated example, not universal thresholds for strong or weak performance.

When reporting results, teams can preserve the distinction between the measured checklist performance and the broader claims they might wish to make about quality. A Content score can support a statement about the presence and completeness of requested elements under the benchmark’s evaluation procedure. It cannot, by itself, establish numerical correctness, visual polish, or transfer to other benchmarks. Keeping the metric-level results alongside Rule and Visual makes those boundaries clearer and reduces the risk of attributing more meaning to an aggregate than the underlying checks support.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.