Publications in reversed chronological order. For a full list including domestic (Japanese) publications, invited talks, awards, and grants, please see the Japanese version of this page.
CONF Conference JOUR Journal
2026
ASE
AI4SE
How Well Do LLMs Generate Taxonomies in the SE Domain? A Multi-Perspective Evaluation Framework
Sota Nakashima, Yuta Ishimoto, Masanari Kondo, Tao Xiao, and Yasutaka Kamei
In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. Accepted, to appear, Oct 2026 arXiv:2608.01592
Taxonomies provide a shared conceptual framework for organizing heterogeneous observations in software engineering (SE) research. Manually constructing such taxonomies is labor-intensive and requires annotators with expertise in the SE domain. While advances in Large Language Models (LLMs) have led to the emergence of automated taxonomy generation methods outside the SE domain, their applicability to technically complex SE artifacts remains unclear. In this experience paper, we present the first comprehensive empirical evaluation of how state-of-the-art automated methods perform on SE artifacts through a multi-perspective evaluation framework, including taxonomy quality, alignment with taxonomies defined by human experts, reliability under independent annotation, and efficiency. To support this evaluation, we systematically collect seven SE papers with publicly available artifacts and human-defined taxonomies, and conduct experiments using two automated methods (TnT-LLM and CLIMB) with five state-of-the-art LLMs. Our evaluation reveals a clear trade-off: TnT-LLM constructs high-quality taxonomies comparable to human-defined ones but incurs substantially higher cost and runtime and tends to generate overly complex taxonomies, whereas CLIMB is 15–40× faster and 8–49× cheaper but tends to score lower on quality when technical inference beyond surface-level similarity is required. These findings suggest that TnT-LLM and CLIMB can be used in practical situations in the SE domain, while researchers should first assess the complexity of the generated taxonomies and their cost using a subset of the target data to decide whether to use automated methods or human experts. Our work represents a first step toward a systematic understanding of automated taxonomy generation in SE, offering actionable insights for future research and practice.
ASE’26 (accepted after major revision, extended work of APSEC’25)
ISSRE
SE4AI
RepTran: Search-Based Repair of Transformer Models
Yuta Ishimoto, Paolo Arcaini, Fuyuki Ishikawa, Masanari Kondo, Naoyasu Ubayashi, and 1 more author
In Proceedings of the 37th International Symposium on Software Reliability Engineering (ISSRE), pp. Accepted, to appear, Oct 2026 arXiv:2607.11193
To ensure the overall quality of AI-enabled software, not only traditional software components but also AI components need to be tested and repaired. Among AI components, Transformer models are increasingly integrated into software systems, which makes their misbehaviors critical. Although prior work in the software engineering community has proposed deep neural network (DNN) repair methods, most overlook Transformer-specific structures. We propose RepTran, a search-based repair method for Transformer models. It targets their feed-forward networks (FFNs), which play a central role in the architecture. RepTran identifies suspicious weights by combining two types of scores: a variance-based neuron score and an existing bidirectional score. It then iteratively optimizes these weights using differential evolution. Our evaluation includes 18 fault benchmarks constructed from CIFAR-100 and Tiny-ImageNet. We compare RepTran against three baselines: random weight selection, Arachne (a state-of-the-art DNN repair method), and ArachneW, which enables Arachne to control the number of selected weights. RepTran achieved an average repair rate of 74.7%, statistically outperforming random selection and Arachne across all benchmarks. Effect size analysis revealed that RepTran achieved higher repair rates than ArachneW regardless of the number of selected weights. These results suggest that RepTran is effective for enhancing the reliability of AI-enabled software.
The reproducibility of software defect datasets is essential for obtaining reliable and comparable research results. Zhu et al. have shown that defect datasets such as Defects4J suffer from reproduction failures (i.e., reported bugs become non-reproducible) as time passes since their creation. However, it remains unclear whether these findings generalize to quantum software defect datasets. We therefore conduct a replication study of the prior work using Bugs4Q, a widely used dataset of real-world bugs in quantum programs. Our analysis includes 77,700 quantum program executions of 37 Bugs4Q artifacts across 21 core-library versions. The experimental results showed that the reproducibility of Bugs4Q dropped from 62.2% on Qiskit v0.20.1 to 16.2% on v2.3.1, the latest version as of April 1, 2026. A manual inspection of the root causes further indicated that 93.6% of the failures were dependency-related. While these findings are consistent with those of the prior work, we also observed differences. In particular, most reproduction failures in Bugs4Q cannot be resolved merely by adjusting dependency versions; instead, they require source-code modifications such as migrating import paths and API invocations. Based on this observation, we curated Bugs4Q-Robust, a patched version of Bugs4Q to restore reproducibility. Bugs4Q-Robust increases reproducibility from 16.2% to 78.4% on Qiskit v2.3.1. Our findings highlight the importance of continuous dataset maintenance in the rapidly evolving quantum software ecosystem.
Agentic coding – software development workflows in which autonomous coding agents plan, implement, and submit code changes with minimal human involvement – is rapidly gaining traction. Prior work has shown that Pull Requests (PRs) produced using coding agents (Agentic-PRs) are accepted less often than PRs that are not labeled as agentic (Human-PRs). The rejection reasons for a single agent (Claude Code) have been explored, but a comparison of how rejection reasons differ between Agentic-PRs generated by different agents has not yet been performed. This comparison is important since different coding agents are often used for different purposes, which can lead to agent-specific failure patterns. In this paper, we inspect 654 rejected PRs from the AIDev dataset covering five coding agents, as well as a human baseline. Our results show that seven rejection modes occur only in Agentic-PRs, including distrust of AI-generated code. We also observe agent-specific patterns (e.g., automated withdrawal of inactive PRs by Devin), reflecting differences in how agents are configured and used in practice. Notably, a large proportion of rejected PRs (67.9%) lack explicit reviewer feedback, making their rejection reasons difficult to determine. To mitigate this issue, we propose a set of heuristics that reduce the proportion of such cases, offering a practical preprocessing step for future studies of PR rejection in agentic coding.
MSR-MC’26 – arXiv
SANER
QSEAI4SE
Leveraging Mutation Analysis for LLM-based Repair of Quantum Programs
Chihiro Yoshida, Yuta Ishimoto, Olivier Nourry, Masanari Kondo, Makoto Matsushita, and 2 more authors
In Proceedings of the 33rd International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 92–97, Mar 2026. DOI: 10.1109/SANER67736.2026.00019arXiv:2601.12273
In recent years, Automated Program Repair (APR) techniques specifically designed for quantum programs have been proposed. However, existing approaches often suffer from low repair success rates or poor understandability of the generated patches. In this study, we construct a framework in which a large language model (LLM) generates code repairs along with a natural language explanation of the applied repairs. To investigate how the contextual information included in prompts influences APR performance for quantum programs, we design four prompt configurations with different combinations of static information, dynamic information, and mutation analysis results. Mutation analysis evaluates how small changes to specific parts of a program affect its execution results and provides more detailed dynamic information than simple execution outputs such as stack traces. Our experimental results show that mutation analysis can provide valuable contextual information for LLM-based APR of quantum programs, improving repair success rates (achieving 94.4% in our experiment) and in some cases also improving the quality of generated explanations. Our findings point toward new directions for developing APR techniques for quantum programs that enhance both reliability and explainability.
SANER-ERA’26
2025
APSEC
AI4SE
How Far Have LLMs Come Toward Automated SATD Taxonomy Construction?
Sota Nakashima, Yuta Ishimoto, Masanari Kondo, Tao Xiao, and Yasutaka Kamei
Technical debt refers to suboptimal code that degrades software quality. When developers intentionally introduce such debt, it is called self-admitted technical debt (SATD). Since SATD hinders maintenance, identifying its categories is key to uncovering quality issues. Traditionally, constructing such taxonomies requires manually inspecting SATD comments and surrounding code, which is time-consuming, labor-intensive, and often inconsistent due to annotator subjectivity. In this study, we investigated to what extent large language models (LLMs) could generate SATD taxonomies. We designed a structured, LLM-driven pipeline that mirrors the taxonomy construction steps researchers typically follow. We evaluated it on SATD datasets from three domains: quantum software, smart contracts, and machine learning. It successfully recovered domain-specific categories reported in prior work, such as Layer Configuration in machine learning. It also completed taxonomy generation in under two hours and for less than $1, even on the largest dataset. These results suggest that, while full automation remains challenging, LLMs can support semi-automated SATD taxonomy construction. Furthermore, our work opens up avenues for future work, such as automated taxonomy generation in other areas.
ESEM-Short’25 – APSEC-ERA’25
EASE
QSE
Evaluating Mutation-based Fault Localization for Quantum Programs
Yuta Ishimoto, Masanari Kondo, Naoyasu Ubayashi, Yasutaka Kamei, Ryota Katsube, and 2 more authors
In Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 666–671, Jun 2025. DOI: 10.1145/3756681.3757022
Quantum computers leverage the principles of quantum mechanics to execute operations. They require quantum programs that define operations on quantum bits (qubits), the fundamental units of computation. Unlike traditional software development, the process of creating and debugging quantum programs requires specialized knowledge of quantum computation, making the development process more challenging. In this paper, we apply and evaluate mutation-based fault localization (MBFL) for quantum programs with the aim of enhancing debugging efficiency. We use quantum mutation operations, which are specifically designed for quantum programs, to identify faults. Our evaluation involves 23 real-world faults and 305 artificially induced faults in quantum programs developed with Qiskit. The results show that real-world faults are more challenging for MBFL than artificial faults. In fact, the median EXAM score, which represents the percentage of the code examined before locating the faulty statement (lower is better), is 1.2% for artificial benchmark and 19.4% for the real-world benchmark in the worst-case scenario. Our study highlights the potential and limitations of MBFL for quantum programs, considering different fault types and mutation operation types. Finally, we discuss future directions for improving MBFL in the context of quantum programming.
With the increasing prevalence of software incorporating deep neural networks (DNNs), quality assurance for these software systems has become a crucial concern. To this end, various methods have been proposed to repair the misbehavior of DNNs by modifying their weights. However, these repair methods may not meet the developer’s needs for a given dataset and model. In this study, we build prediction models for repair outcomes (i.e., repairs and breaks) to help determine whether the repair method is likely to work. By using our prediction models, developers and operators of DNNs can decide whether or not to apply a repair method, and if so, which method to use. Our prediction models utilize four metrics as explanatory metrics that represent the confidence or ambiguity in the DNN predictions. We experimented with four repair methods and 10 datasets. The experimental results demonstrate that our prediction models successfully select a repair method that meets developers’ needs in 16 out of 24 cases, resulting in an average time saving of 16.29% compared to the naive method. Based on these results, our prediction models can reduce costs for developers and operators when deciding whether to employ repair methods for real-world applications of DNNs.
TOSEM (extended work of CAIN’23)
2024
APSEC
QSEMSR
An Empirical Study on Self-Admitted Technical Debt in Quantum Software
Yuta Ishimoto, Yuto Nakamura, Ryota Katsube, Naoto Sato, Hideto Ogawa, and 3 more authors
In Proceedings of the 31st Asia-Pacific Software Engineering Conference (APSEC), pp. 41–50, Dec 2024. DOI: 10.1109/APSEC65559.2024.00015
Quantum computers, which utilize the principles of quantum mechanics, are expected to be applied to a wide range of fields. With the advancement of quantum computer development, a lot of quantum software, which enables the operation of quantum computers, has been developed. It has a distinct nature (e.g., superposition and entanglement of qubits) compared to traditional software, leading to the unique challenges of its development. While prior studies have clarified and defined some unique challenges of quantum software, many remain unclear due to limited research. In this study, we conducted an empirical study of Self-Admitted Technical Debt (SATD) for quantum software. SATD is a type of technical debt, a problem in the code that the developer is aware of. Hence, we conjecture that analyzing SATDs can reveal the unique challenges developers face when developing quantum software. We manually coded 202 comments from the Python files of the 61 open-source quantum software on GitHub. The 202 comments correspond to a 95% confidence level with a 5% confidence interval, as in previous studies. The results showed that 88 comments (45.6% of all SATD comments) were quantum-specific SATDs (QSATDs), which require knowledge of quantum computation to repay. Furthermore, we propose a taxonomy for QSATDs. This taxonomy, which consists of four main categories and eight subcategories, classifies QSATDs in terms of quantum-specific aspects such as circuit implementation, backend, and algorithms. Our empirical results are beneficial for quantum software developers, helping them understand implementation areas that require attention. For researchers, our results promote further research, including the exploration of challenges in QSATD repayment.
ICSME’24 – APSEC’24
2023
QRS-C
SE4AI
Uncertainty-aware Metamorphic Testing for Robust Object Detection Models
Jianhong Wang, Yuta Ishimoto, Masanari Kondo, Yasutaka Kamei, and Naoyasu Ubayashi
In Proceedings of the 23rd International Conference on Software Quality, Reliability, and Security Companion (QRS-C), pp. 9–17, Oct 2023. DOI: 10.1109/QRS-C60940.2023.00058
Object detection models are widely used in safety-critical systems in industrial fields such as autonomous driving. It is essential to improve the robustness of object detection models capable of avoiding failures in realistic scenarios. This study proposes metamorphic testing on the object detection model robustness. The metamorphic testing generates metamorphic tested images that simulate the realistic perturbations in production environments. In order to evaluate the model robustness, Bayesian uncertainty is used to describe how the model maintains its reliability on the prediction. This paper shows how metamorphic testing performs the evaluation of the model robustness with Bayesian uncertainty, and how metamorphic testing can improve the model robustness by retraining the models.
Context: If deep learning models in safety-critical systems misbehave, serious accidents may occur. Previous studies have proposed approaches to overcome such misbehavior by detecting and modifying the responsible faulty parts in deep learning models. For example, fault localization has been applied to deep neural networks to detect neurons that cause misbehavior. Objective: However, such approaches are not applicable to deep learning models that have internal states, which change dynamically based on the input data samples (e.g., recurrent neural networks (RNNs)). Hence, we propose a new fault localization approach to be applied to RNNs. Methods: We propose probabilistic automaton-based fault localization (PAFL). PAFL enables developers to detect faulty parts even in RNNs by computing suspiciousness scores with fault localization using n-grams. We convert RNNs into probabilistic finite automata (PFAs) and localize faulty sequences of state transitions on PFAs. To consider various sequences and to detect faulty ones more precisely, we use n-grams inspired by natural language processing. Additionally, we distinguish data samples related to the misbehavior to evaluate PAFL. We also propose a novel suspiciousness score, average n-gram suspiciousness (ANS) score, based on n-grams to distinguish data samples. We evaluate PAFL and ANS scores on eight publicly available datasets on three RNN variants: simple recurrent neural network, gated recurrent units, and long short-term memory. Results: The experiment demonstrates that ANS scores identify faulty parts of RNNs when n is greater than one. Moreover, PAFL is statistically significantly better and has large effect sizes compared to state-of-the-art fault localization in terms of distinguishing data samples related to the misbehavior. Specifically, PAFL is better in 66.74% of the experimental settings. Conclusion: The results demonstrate that PAFL can be used to detect faulty parts in RNNs. Hence, in future studies, PAFL can be used as a baseline for fault localization in RNNs.
ICSME’21 – IST
CAIN
SE4AI
An Initial Analysis of Repair and Side-effect Prediction for Neural Networks
Yuta Ishimoto, Ken Matsui, Masanari Kondo, Naoyasu Ubayashi, and Yasutaka Kamei
In Proceedings of the 2nd International Conference on AI Engineering – Software Engineering for AI (CAIN), pp. 80–85, May 2023. DOI: 10.1109/CAIN58948.2023.00017
With the prevalence of software systems adopting neural network models, the quality assurance of these systems has become crucial. Hence, various studies have proposed repairing methods for neural network models so far to improve the quality of the models. While these methods are evaluated by researchers, it is difficult to tell whether they succeed in all models and datasets (i.e., all developers’ environments). Because these methods require many resources, such as execution times, failing to repair neural networks would cost developers their resources. Hence, if developers can know whether repairing methods succeed before adopting them, they could avoid wasting their resources. This paper proposes prediction models that predict whether repairing methods succeed in repairing neural networks using a small resource. Our prediction models predict repairs and side-effects of repairing methods, respectively. We evaluated our prediction models on a state-of-the-art repairing method Arachne on three datasets, Fashion-MNIST, CIFAR-10, and GTSRB, and found our prediction models achieved high performance, an average ROC-AUC of 0.931 and an average f1-score of 0.880 for the side-effects and an average ROC-AUC of 0.768 and an average f1-score of 0.725 for the repairs.
Issue reports are a pivotal interface between developers and users for receiving information about bugs in their products. In practice, issue reports often have incorrect information or insufficient information to enable bugs to be reproduced, and this has the effect of delaying the entire bug-fixing process. To facilitate their bug-reproduction work, GitHub has provided a new feature that allows users to share videos (e.g., mp4 files). Using such videos, reports can be made to developers about the details of bugs by recording the symptoms, reproduction steps, and other important aspects of bug information. While such visual issue reports have the potential to significantly improve the bug-fixing process, no studies have empirically examined this impact. In this paper, we conduct a preliminary study to identify the characteristics of visual issue reports by comparing them with non-visual issue reports. We collect 1,230 videos and 18,760 images from 226,286 issues on 4,173 publicly available repositories. Our preliminary analysis shows that issue reports with images are described in fewer words than non-visual issue reports. In addition, we observe that most discussions in visual issue reports are concerned with either conditions for reproduction (e.g., when) or GUI (e.g., pageviewcontroller).
