Japan PMDA Opens Consultation on Review Points for Medical Devices Using Generative Artificial Intelligence: Comments Due 17 March 2027

Japan PMDA Opens Consultation on Review Points for Medical Devices Using Generative Artificial Intelligence

Published: 24 October 2026 19:34 | Category: Technical Guidance

On 14 September 2026, the Center for Product Evaluation of Japan's Pharmaceuticals and Medical Devices Agency (PMDA) prepared a 29-page document entitled Review Points for Medical Devices Utilizing Generative Artificial Intelligence (Draft). The Japanese draft was posted on the PMDA website on 18 September 2026 and public comments opened the same day. The comment period runs from 18 September 2026 to 17 March 2027. An English reference translation is marked as the October 2026 version, and PMDA posted the link in its What's New section on 14 October 2026. The document states that the English version is for reference only and that the Japanese original prevails in the event of any inconsistency.

This point should be clear at the outset: the document is still a draft. The review points are not regulatory provisions and do not constitute approval criteria for any product. The final version will be published separately after the comment period ends. What companies can use now is the review approach and the evidence methods described in the draft.

Scope: Which Products Are Covered

The draft defines generative artificial intelligence narrowly. It refers to systems that can respond to broad questions through natural language text or voice input and generate content not dependent on a specific task. In practice, these are large language models (LLMs) or vision-language models (VLMs). Traditional discriminative models, such as those that detect a type of abnormality in an image, are not within this scope and remain subject to existing review pathways.

The draft covers generative artificial intelligence medical devices, meaning systems whose core functions are implemented by generative models and which constitute medical devices under the Pharmaceuticals and Medical Devices Act (PMD Act). The draft also breaks these systems into components: the model itself, the user interaction interface, guardrails for processing inputs and outputs, and the external tool set used for retrieving information and executing processing. This breakdown has practical significance. In a submission, which components are self-developed and which come from third parties will directly determine what evidence must be submitted.

Five Technical Characteristics and Five Review Concerns

The draft summarizes the technical characteristics of generative artificial intelligence under five headings and attaches review considerations to each.

• Large-scale training and model architecture. Model capability improves with the scale of parameters and data, and development and operation require substantial computing power. Most device manufacturers will use external foundation models and perform system integration and interface optimization themselves. This creates concerns: training data details may not be disclosed to developers, root causes may be difficult to identify, dataset bias may be masked, and it may be difficult to explain whether test data overlap with training data, that is, data leakage.
• Context-dependent generation. The same input instruction may produce different outputs, and outputs may change with context such as conversation history. Hallucinations therefore exist, meaning information that appears credible but is untrue or contrary to fact. Subtle differences in instruction wording can change results, reducing repeatability. Accumulated dialogue history can also cause model behavior to drift gradually.
• Generalization and multi-purpose use. One model can simultaneously perform question answering, summarization, translation and multi-disease diagnostic support, making functional boundaries unclear. Tasks outside the approved use may also be executed. The draft therefore requires guardrails to prevent off-label use, use contrary to precautions and misuse.
• Interoperability with external systems. Retrieval-augmented generation (RAG) and calls to external tools make the device dependent on external data sources and systems. Changes to cited data, deletion of data or failure of connected systems may alter device performance. It is therefore necessary to define the controlled scope, versions and availability, and to establish monitoring mechanisms.
• Action autonomy. Agentic functions can decompose tasks, make plans, call tools, evaluate results and decide next steps. The degree of human involvement ranges from step-by-step approval to observation only. The higher the autonomy, the greater the risk of bypassing constraints set by humans, taking shortcuts to achieve goals through reward hacking, and amplifying hallucinations and out-of-scope tasks along the action chain. The draft accordingly requires design features for human confirmation, stopping and intervention, and an auditable record of the execution sequence.

How to Demonstrate Effectiveness: Usefulness and Product Performance

Under Japan's current review logic, effectiveness is divided into two parts: usefulness, meaning contribution to clinical outcomes, and product performance, meaning performance under specified conditions. The draft continues this framework but shifts the focus of evidence.

For usefulness, the draft provides two pathways. One is to demonstrate improvement in clinical outcomes directly. The other is to show that product performance results are sufficient to support clinical significance. Which pathway to use will be decided case by case based on the sufficiency of product data and the clinical positioning. If prospective evaluation is required, a clinical trial will be necessary. For diagnostic support products not limited to a specific disease, the draft notes that it is difficult to design a trial protocol covering all use contexts. It suggests considering approval first for a narrower indication and then expanding claims gradually.

For product performance, the draft requires assessment at three layers: general language capability and instruction following, foundational medical knowledge, and task-specific performance. The purpose of this layered approach is to avoid using one broad accuracy figure to conceal capability gaps.

How to Test Performance: Property-Based Testing and Three Judgment Methods

This is the part of the draft that merits close reading. Outputs from generative models fluctuate, and many tasks do not have a single correct answer. The draft therefore does not recommend merely comparing a specific numerical value. Instead, it first defines the properties that the target behavior should maintain, then uses property-based testing to search a large automatically generated test space for inputs and operations that violate those properties, thereby providing statistical assurance.

The specific approach is to test the semantic relationship between input and output, that is, metamorphic relations: after paraphrasing, adding or removing conditions, or reordering references, does the output remain consistent or change in a logically corresponding way? This is combined with noise tolerance testing: insert irrelevant text, typographical errors, homophones or jailbreak-style inputs and observe whether the output still holds the original boundaries. The draft requires five properties to be maintained: text accuracy, contextual appropriateness, instruction following, safety boundaries, and reproducibility and stability. Specific test methods are provided for each.

The draft lists three ways to judge whether an output is appropriate: automatic judgment based on empirical rules, expert evaluation by physicians and other specialists according to the clinical context, and scoring by another language model against a predefined rubric, known as LLM-as-a-Judge. The choice or combination depends on whether the task has a standard answer and on the scale of evaluation.

The role of benchmarks is deliberately limited. Scores such as JGLUE, Japanese MT-Bench, Rakuda and IgakuQA can be used to understand a baseline, but benchmark performance is not directly linked to clinical relevance. If a model has seen the test set, scores may also be overestimated. The draft states clearly that benchmark scores alone are insufficient and that an overall judgment must consider whether the test set is appropriate, the evaluation process and methods, and the study protocol.

Post-Market Is Not a Supplement but Half of the Evaluation

The draft acknowledges that pre-market evaluation cannot realistically cover long-tail situations in real-world use and that outputs may drift as external reference information changes. It therefore proposes a total lifecycle approach. Before launch, effectiveness, performance and the ability to maintain performance over time and under different conditions should be evaluated under representative conditions. After launch, performance, adverse events and hallucinations should be actively collected, with periodic reporting on whether these findings raise clinical concerns. Even if clinical outcomes are acceptable, agentic functions may have problems in the execution process, so execution logs should also be included in monitoring. The draft also requires that the handling of problem events be agreed in advance with the reviewer.

The document refers to Japan's existing IDATEN framework, a mechanism for machine learning variability that allows pre-specified updates within the approved scope. However, the draft also notes that for products that change in real time, IDATEN has limited coverage, and rebalancing before and after launch is still necessary.

Human Factors: Automation Bias Addressed Separately

The draft compares generative model outputs item by item with medical literature. Literature has authors, journals and verifiable citations; generative output without RAG has unclear sources. Literature has accumulated peer review and society assessment; model output has none. Physicians are trained in critical reading, whereas algorithmic output may appear objective, accurate and easy to accept, making automation bias likely.

The conclusion focuses on the use environment: the device should be used where users can correctly understand the uncertainty and reliability of outputs. Design should present supporting evidence, indicate information boundaries, and clarify the target population and failure conditions. The draft also states that residual incompleteness in outputs is not in itself a reason to refuse approval. The key is to give users the conditions to make judgments.

Timeline and Submission of Comments

The draft was published on 18 September 2026, and comments are due by 17 March 2027. Comments should be submitted in Japanese to PMDA's Medical Devices Review Department. PMDA has indicated that when the final review points are published, it will also publish the comments received and the corresponding handling results.

Sources and Verification Note

• Pharmaceuticals and Medical Devices Agency (PMDA), Review Points for Medical Devices Utilizing Generative Artificial Intelligence (Draft), 14 September 2026, English reference translation October 2026 version. https://www.pmda.go.jp/files/000283182.pdf
• PMDA, Program Medical Devices review points consultation page in Japanese, stating that the draft on medical devices utilizing generative artificial intelligence was published on 18 September 2026 and that the comment period runs from 18 September 2026 to 17 March 2027. https://www.pmda.go.jp/review-services/drug-reviews/about-reviews/devices/0058.html
• PMDA, What's New page, entry dated 14 October 2026 regarding publication of the English translation of the draft review points. https://www.pmda.go.jp/english/0006.html

Verification scope and access date: The three official PMDA pages above were accessed on 24 October 2026, and the full 29-page English reference translation was read. The draft's scope, technical characteristics, evaluation principles, post-market measures and consultation timeline were checked item by item. All dates, clause locations and framework names in this article are taken from the original texts above; no secondary reports were cited.

Applicability note: This article is an interpretation of a public PMDA draft. The draft is not final and does not constitute a conclusion or commitment regarding any specific product, company or registration application, nor does it represent the final position of PMDA or Japanese regulators. The review points are not regulations. Actual review will be based on PMDA individual consultations and judgments made at the time of formal application. The text may also change with comments and subsequent updates; please refer to the official original.

AIMEILI Regulatory Interpretation

Status and weight. The document is a draft, not a regulation or an approval standard. It should be treated as an early signal of PMDA's evidentiary expectations rather than a binding checklist. Manufacturers can use it for gap assessment and for dialogue with PMDA, but should not cite it as a legal requirement.

Key regulatory shift. PMDA is separating generative-model devices from traditional discriminative devices and focusing on two questions: how usefulness is demonstrated and how product performance is measured. The draft accepts that outputs vary and that many tasks lack a single correct answer. It therefore favors property-based testing and lifecycle evidence over a single accuracy figure. For companies, this means the evidence package must explain why the model is useful, how it performs under defined conditions, and how performance is maintained after launch.

Third-party model dependency. Many device makers will integrate external foundation models. The draft recognizes that training data may not be disclosed, root causes may be hard to locate, and data leakage may be difficult to rule out. Companies should secure contractual access to model documentation, version control, change notifications and testing. The boundary between self-developed and third-party components will shape the evidence required.

Post-market and human factors. PMDA treats post-market surveillance as part of the evaluation, not as a supplement. Hallucinations, performance drift, adverse events and agentic execution logs should be collected. Human factors and automation bias are also central: user interface, labeling and instructions must help users understand uncertainty, reliability, information boundaries, target population and failure conditions.

Business Impact

• Evidence planning. Budget for property-based testing, metamorphic and noise testing, expert clinical review and predefined scoring rubrics. Do not rely on benchmark scores alone.
• Supplier and contract management. For third-party foundation models, negotiate access to training-data summaries, model versions, change notices, evaluation data and incident information. Define responsibility for post-market updates.
• Clinical and indication strategy. For disease-agnostic diagnostic support, consider a narrower initial indication and a staged claim expansion plan rather than a broad first filing.
• Post-market system build. Establish intake and triage for hallucinations, adverse events, performance drift and agentic execution logs. Define periodic reporting and pre-agreed handling with the regulator.
• Human factors and labeling. Prepare user interface and labeling evidence on uncertainty, reliability, information boundaries, target users and failure conditions. Address automation bias explicitly.
• Regulatory timeline. Comments are due 17 March 2027, and the final text is pending. Engage PMDA consultation early, monitor the official Japanese text, and treat the English translation as reference only.
• Distributors and CROs. Distributors should strengthen supplier due diligence. CROs should validate test methods and clinical evaluation designs. Regulatory affairs teams should run gap assessments against the draft's five technical characteristics.

Overall, the draft is likely to raise the bar for documentation, testing and post-market evidence for generative-model devices in Japan. Companies that prepare now can reduce later review risk and align global evidence packages with PMDA expectations, while keeping in mind that the final review points may change after consultation.