Prompt Augmentation with Context at Repository Scale for Code Summarization with the Cache-Augmented Generation Paradigm
DOI:
https://doi.org/10.59261/jequi.v8i5.472Keywords:
Code Summarization, Large Language Models, Prompt Augmentation, Cache-Augmented Generation (CAG), Retrieval-Augmented Generation (RAG)Abstract
Background: Code summarization is a fundamental aspect of software development and maintenance. However, the performance of large language models (LLMs) on this task can still be improved through prompt augmentation techniques, including Cache-Augmented Generation (CAG), which remains largely unexplored in the context of repository-level code.
Objective: This study systematically and empirically evaluates the effectiveness of CAG for code summarization when repository-level code structure is provided as context. Its performance is compared with that of Retrieval-Augmented Generation (RAG) and a base model.
Methods: This quantitative experimental study used the Llama 3.2 3B Instruct model on a sample of the CodeSearchNet dataset grouped by Python project repository. The three methods were evaluated and compared using three metrics: Summary-level Semantic Similarity (SIDE), Time to First Token (TTFT), and BERTScore.
Results: CAG achieved the lowest reported TTFT and a marginally higher BERTScore F1 score, whereas RAG achieved the highest SIDE score. Specifically, RAG attained a mean SIDE score of 0.596 and a mean BERTScore F1 score of 0.801. CAG achieved a mean TTFT of 0.188 seconds, considerably faster than RAG (2.680 seconds) and the base model (1.377 seconds).
Conclusion: Preloading comprehensive code context into the LLM's context window is effective for producing semantically aligned code summaries with minimal latency. However, CAG has a notable limitation: its cache must be updated when new information needs to be incorporated.
Downloads
References
Agrawal, R., & Kumar, H. (2025). Enhancing Cache-Augmented Generation (CAG) with Adaptive Contextual Compression for Scalable Knowledge Integration. ArXiv.
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. T. (2024). Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization). Proceedings - International Conference on Software Engineering, 2720–2732. https://doi.org/10.1145/3597503.3639183
Dhulshette, N., Shah, S., & Kulkarni, V. (2025). Hierarchical Repository-Level Code Summarization for Business Applications Using Local LLMs. http://arxiv.org/abs/2501.07857
Grattafiori, A. (2024). The llama 3 herd of models."arXiv preprint arXiv:2407.21783.
Gupta, S., Ranjan, R., & Singh, S. N. (2024). A comprehensive survey of retrieval-augmented generation (rag): Evolution, current landscape and future directions. arXiv preprint arXiv:2410.12837.
Huang, H.-H., Chan, B. J., Chen, C.-T., & Cheng, J.-H. (2024). Don’t Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks. https://doi.org/10.1145/3701716.3715490
Lester, B., Al-Rfou, R., & Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. http://arxiv.org/abs/2104.08691
Lomshakov, V., Podivilov, A., Savin, S., Baryshnikov, O., Lisevych, A., & Nikolenko, S. (2024, November). ProConSuL: Project context for code summarization with LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track (pp. 866-880).
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., Gong, M., Zhou, M., Duan, N., Sundaresan, N., … Liu, S. (2021). CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. http://arxiv.org/abs/2102.04664
Luo, Q., Ye, Y., Liang, S., Zhang, Z., Qin, Y., Lu, Y., Wu, Y., Cong, X., Lin, Y., Zhang, Y., Che, X., Liu, Z., & Sun, M. (2024). RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generation. EMNLP 2024 - 2024 Conference on Empirical Methods in Natural Language Processing, Proceedings of System Demonstrations. https://doi.org/10.18653/v1/2024.emnlp-demo.46
Lyu, M. R., Ray, B., Roychoudhury, A., Tan, S. H., & Thongtanunam, P. (2025). Automatic programming: Large language models and beyond. ACM Transactions on Software Engineering and Methodology, 34(5), 1-33.
Makharev, V., & Ivanov, V. (2025). Code Summarization Beyond Function Level. http://arxiv.org/abs/2502.16704
Mastropaolo, A., Ciniselli, M., Di Penta, M., & Bavota, G. (2023). Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization. http://arxiv.org/abs/2312.15475
Parvez, M. R., Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K.-W. (2021). Retrieval Augmented Code Generation and Summarization. http://arxiv.org/abs/2108.11601
Su, C.-Y., Bansal, A., Huang, Y., Li, T. J.-J., & McMillan, C. (2024). Context-aware Code Summary Generation. http://arxiv.org/abs/2408.09006
Tingting Xu, Yun Miao, Chunrong Fang, Hanwei Qian, Xia Feng, Zhenpeng Chen, Chong Wang, Jian Zhang, Weisong Sun, Zhenyu Chen, & Yang Liu. (2023). A Prompt Learning Framework for Source Code Summarization. 2022-March, 36–47. https://doi.org/10.1145/nnnnnnn.nnnnnnn
Wang, Y., Xie, C., Zhang, W., & Qiu, X. (2025, July). FRCS-LLM: A Framework for Refining Code Summarization in Large Language Models via Pre-trained Models. In 2025 IEEE 49th Annual Computers, Software, and Applications Conference (COMPSAC) (pp. 1244-1253). IEEE.
Yang, Z., Chen, S., Gao, C., Li, Z., Hu, X., Liu, K., & Xia, X. (2025). An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities. ACM Transactions on Software Engineering and Methodology, 34(7). https://doi.org/10.1145/3717061
Zhang, Q., Fang, C., Xie, Y., Zhang, Y., Yu, S., Sun, W., ... & Chen, Z. (2026). A survey on large language models for software engineering. Science China Information Sciences, 69(4), 141102.
Zhu, Q., Luo, X., Liu, F., Gao, C., & Che, W. (2024). A Survey on Natural Language Processing for Programming. 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024 - Main Conference Proceedings. https://doi.org/10.63317/2tdeieb24zmy
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Erwin Hartanto, Maria Seraphina Astriani
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International (CC-BY-SA). that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
