Multimodal safety evaluation

Language- and culture-aware safety evaluation across text, speech, images, video, retrieval, and model-mediated actions. In practice, the work is bounded by a defined product or model decision, named audiences and locales, representative inputs, and acceptance criteria that can be reviewed.

Safety controls trained in a few languages may miss euphemism, dialect, code-switching, visual context, speech prosody, or harmful tool behavior that becomes visible only across modalities and turns. The useful starting point is the smallest representative flow that can expose the cause, impact, and ownership of the problem.

Build the threat model: Identify capabilities, users, locales, cultural contexts, modalities, abuse paths, protected groups, and decision thresholds. Create representative challenges: Develop multilingual and multimodal cases with calibrated rubrics, adversarial variants, and human-review guidance. Test mitigations in system: Measure detection and refusal quality, overblocking, cross-turn drift, tool safety, recovery, and residual risk.

The most useful inputs are the product task and user journey, candidate models or system access, priority languages and communities, policies and risk thresholds, representative prompts, media, tools, and and expected outcomes. Admas can begin with a partial package, but missing context, rights, access, owners, or acceptance criteria will be made visible in the plan rather than treated as harmless assumptions.

Typical outputs include multilingual multimodal threat model, safety challenge set and evaluator protocol, segmented mitigation results, and residual-risk register and release recommendation. Deliverables are adapted to the team that must use them, with decisions, evidence, limitations, owners, and next actions made explicit.

Quality is measured against the real task and risk. Relevant evidence can include task success by language and scenario, human-rated meaning and usefulness, safety and policy performance, retrieval and citation fidelity, tool-call correctness, and regressions and disparities hidden by aggregate scores. Sampling, severity rules, reviewers, adjudication, and pass or fail thresholds should be agreed before the result is used as a release decision.

Models and automated checks can generate candidates, expand test sets, cluster failures, and accelerate analysis. Qualified humans define what good means, identify culturally or linguistically plausible failures, adjudicate close cases, and own consequential release judgments. The right allocation depends on consequence, content stability, available references, language coverage, reversibility, and the cost of a plausible but wrong result.

The estimate changes with number of languages, modalities, systems, and scenarios, risk level, dataset creation needs, evaluator specialization, sampling and adjudication depth, and experiment and reporting cadence. Pricing should distinguish setup and discovery, repeatable units, specialist or engineering time, independent review, management, and external costs. A low unit price is not comparable if it excludes the QA cycle or shifts rework back to the buyer.