Projects
Context-Aware Mapping of 2D Drawing Annotations to 3D CAD Features Using LLM-Assisted Reasoning for Manufacturing Automation
Publication: Khan, M.T., Chen, L., Feng, W., Moon, S.K., 2026. Context-Aware Mapping of 2D Drawing Annotations to 3D CAD Features Using LLM-Assisted Reasoning for Manufacturing Automation. IEEE International Conference on Industrial Engineering and Applications (IEEE ICIEA 2026). Under Review.
Overview: This paper presents a deterministic-first, context-aware framework that maps 2D drawing entities to 3D CAD features to produce a unified manufacturing specification. The system combines semantic enrichment via VLMs, interpretable correspondence scoring with engineering-aware heuristics, and controlled LLM-assisted escalation to resolve contextual ambiguity in drawing annotations.
Methodology: A multi-stage pipeline: (1) VLM-based semantic enrichment of raw drawing callouts into structured descriptors; (2) Deterministic scoring combining type compatibility, tolerance-aware dimensional agreement, and context consistency; (3) Engineering-aware heuristic refinement for domain-specific rules; (4) Controlled escalation to multimodal and constrained LLM reasoning for ambiguous cases; (5) Post-fusion human-in-the-loop validation.
Key Contributions
- First study to automatically and contextually map 2D drawing information to 3D CAD features using AI-assisted reasoning with transparent, auditable decision-making
- Deterministic-first escalation framework combining interpretable scoring with controlled LLM-assisted reasoning
- Unified manufacturing specification output binding drawing constraints to CAD features with explicit provenance and confidence
- Results: Mean Precision = 83.67%, Recall = 90.46%, F1 = 86.29% on 20 real CAD-drawing pairs
- Ablation study confirming each pipeline component contributes meaningfully to overall accuracy
Generative AI-Powered Multi-Agent System for Manufacturing Process Planning and Decision-Making
Description: Manual cost estimation, tool selection, and quotation are often time-consuming, labor-intensive, and inconsistent. This project proposes a multi-agent system powered by LLMs and VLMs to automate these processes:
- 2D Engineering drawing information parsing
- 3D CAD model feature recognition
- Retrieval of manufacturing knowledge (via RAG)
- Grounded reasoning for quotation, cost estimation, and process planning
From Drawings to Decisions: A Hybrid Vision-Language Framework for Parsing 2D Engineering Drawings into Structured Manufacturing Knowledge
.png)
Publication: Khan, M.T., Chen, L., Yong, Z., Tan, J.M., Feng, W., Moon, S.K., 2026. From Drawings to Decisions: A Hybrid Vision-Language Framework for Parsing 2D Engineering Drawings into Structured Manufacturing Knowledge. Robotics and Computer-Integrated Manufacturing, 99, 103186.
Overview: This work presents a comprehensive hybrid vision-language framework that automatically parses 2D engineering drawings into structured manufacturing knowledge. The system combines state-of-the-art object detection (YOLOv11-obb) with vision-language models (Donut, Florence-2) to extract manufacturing annotations including GD&T symbols, dimensions, tolerances, materials, surface roughness, threads, notes, and title blocks.
Methodology: A two-stage pipeline: (1) YOLOv11-obb detects and localizes annotation regions in oriented bounding boxes; (2) Extracted image patches are processed by fine-tuned Donut or Florence-2 transformers to generate structured JSON output. The framework eliminates traditional OCR limitations and directly maps visual features to semantic manufacturing information.
Key Contributions
- Hybrid detection + parsing architecture achieving state-of-the-art performance on engineering drawings
- Comprehensive evaluation of OCR-free transformers (Donut, Florence-2) for manufacturing annotation extraction
- Nine annotation categories: GD&T, dimensions, tolerances, materials, notes, radii, surface roughness, threads, title blocks
- JSON-structured output ready for downstream manufacturing systems
- Results: Donut outperforms Florence-2 with Precision = 89.2%, Recall = 99.2%, F1-score 94.0%, and Hallucination rate = 10.8%
- Real-world industrial case study demonstrating manufacturability assessment and downstream applications
Large Language Model Powered Decision Support for a Metal Additive Manufacturing Knowledge Graph
Publication: Khan, M.T., Chen, L., Feng, W., Moon, S.K., 2025. Large Language Model Powered Decision Support for a Metal Additive Manufacturing Knowledge Graph. 11th International Conference of Asian Society for Precision Engineering and Nanotechnology (ASPEN 2025).
Overview: This research introduces a novel LLM powered decision support system built on a comprehensive metal additive manufacturing knowledge graph. The system enables natural language querying of manufacturing expertise, facilitating informed decision-making for material selection, process parameter optimization, and quality control in metal AM operations.
Methodology: The framework integrates a structured knowledge graph capturing relationships between materials, process parameters, and post-processing requirements with an LLM for natural language understanding and generation. A conversational AI interface allows engineers to query manufacturing knowledge through intuitive dialogue.
Key Contributions
- Comprehensive metal additive manufacturing knowledge graph with domain-specific ontology
- LLM-powered natural language interface for manufacturing knowledge retrieval
- Context-aware decision support for material selection and process optimization
- Integration of structured knowledge with unstructured manufacturing literature
- Interactive chatbot demonstrating real-time query resolution and expert recommendations
Automated Parsing of Engineering Drawings for Structured Information Extraction Using a Fine-tuned Document Understanding Transformer
Publication: Khan, M.T., Yong, Z., Chen, L., Tan, J.M., Feng, W., Moon, S.K., 2025. Automated Parsing of Engineering Drawings for Structured Information Extraction Using a Fine-tuned Document Understanding Transformer. IEEE International Conference on Industrial Engineering and Engineering Management (IEEM 2025).
Methodology: A hybrid deep learning approach integrating YOLOv11-obb detection with an OCR-free document understanding transformer (Donut). It extracts nine categories of manufacturing annotations and converts them into structured JSON.
Key Contributions
- Unified detection + parsing model
- Nine key annotation categories: GD&T, tolerances, dimensions, materials, notes, radii, surface roughness, threads, and title blocks
- JSON output for digital manufacturing pipelines
- Results: Precision 94.77% (GD&T), F1 = 97.3%, Hallucination rate = 5.23%
Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction
Publication: Khan, M.T., Chen, L., Ng, Y., Feng, W., Tan, N.Y.J., and Moon, S.K., 2025. Fine-Tuning Vision-Language Model for Automated Engineering Drawing Information Extraction. 9th International Conference on Innovation in Artificial Intelligence (ICIAI 2025).
Methodology: A novel approach combining full parameter fine-tuning (Microsoft’s Florence-2) and zero-shot learning (GPT-4o, Claude-3.5-Sonnet) for vision-language models (VLMs). The system processes engineering drawing images paired with text queries to extract geometric dimensioning and tolerancing (GD&T) information in structured formats.
Key Contributions
- Engineering Drawing VQA Dataset: Curated dataset with 400 annotated engineering drawings for GD&T information extraction
- Comparative analysis of fine-tuning vs. zero-shot VLM approaches for technical drawing understanding
- Multi-model evaluation: Florence-2 (0.23B), GPT-4o, and Claude-3.5-Sonnet
- Results: Florence-2 achieves a 29.95% increase in precision, a 37.75% increase in recall, a 52.40% improvement in F1-score, and a 43.15% reduction in hallucination rate compared to the best-performing closed-source model
Leveraging Vision-Language Models for Manufacturing Feature Recognition in Computer-Aided Designs
Publication: Khan, M. T., Chen, L., Ng, Y. H., Feng, W., Tan, N. Y. J., and Moon, S. K. Leveraging Vision-Language Models for Manufacturing Feature Recognition in Computer-Aided Designs. ASME Journal of Computing and Information Science in Engineering. October 2025; 25(10): 104501.
Overview: This research introduces a VLM approach for automated feature recognition (AFR) in CAD designs, addressing the limitations of traditional rule-based and deep learning methods. The system leverages multi-view isometric images and natural language processing to identify manufacturing features and provide design feedback without requiring extensive geometric computations or large labeled datasets.
Methodology: The framework employs state-of-the-art VLMs including GPT-4o, Claude-3.5-Sonnet, and Llava to process multi-view CAD images and extract manufacturing features through visual question answering (VQA). The approach integrates feature recognition with manufacturability assessment.
Key Contributions
- MFCAD-VLM Dataset: Comprehensive dataset with STEP files, expert-annotated ground truth JSON files, and multi-view isometric images across three complexity levels (easy, medium, hard)
- Prompt-driven VLM-based AFR framework that eliminates the need for complex geometric algorithms or extensive training data
- Multi-model evaluation comparing open-source and closed-source (GPT-4o, Claude-3.5-Sonnet) VLMs
- Results: Achieves high accuracy in feature type recognition and feature quantity prediction
MFCAD-VLM Dataset
The study utilizes the MFCAD-VLM dataset, a comprehensive collection designed to advance research in CAD and AFR. The dataset includes:
- STEP Files: CAD models in STEP format, representing various parts with distinct manufacturing features, categorized by complexity levels (easy, medium, and hard)
- Ground Truth JSON Files: Expert-annotated JSON files corresponding to each STEP file, detailing manufacturing feature types, quantities, and specifics essential for accurate AFR assessment
- Multi-View Isometric Images: Three isometric-view snapshots per CAD model, generated via Python, capturing different viewing angles to aid feature recognition tasks
Automatic Feature Recognition and Dimensional Attributes Extraction from CAD Models for Hybrid Additive-Subtractive Manufacturing
Citation: Khan, M.T., Feng, W., Chen, L., Ng, Y.H., Tan, N.Y.J., Moon, S.K. 2024. Automatic Feature Recognition and Dimensional Attributes Extraction From CAD Models for Hybrid Additive-Subtractive Manufacturing. Proceedings of the ASME 2024 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. Volume 2A: 44th Computers and Information in Engineering Conference (CIE)., Washington, DC, USA. V02AT02A035.
Overview: A deep learning based workflow that automatically recognizes CAD features and extracts dimensional attributes for hybrid additive–subtractive manufacturing.
Methodology:
- Converts CAD STEP models into B-Rep hierarchical graphs and mini-batches for Graph Convolutional Neural Network (GCNN) processing
- Trains a GCNN model on hybrid features; applies to test CAD models for feature labels and geometric dimensions
- Outputs feature types (with counts), stock sizes, and orientations
Key Contributions
- GCNN-based AFR that fuses geometric topology (B-Rep graphs) with synthetic labeled CAD data containing hybrid features
- Automated dimensional extraction (geometric size, stock sizes, and feature orientation)
- End-to-end pipeline supporting hybrid additive–subtractive planning with minimal manual setup
HybridCAD++ Dataset
- 161,441 CAD models with 36 hybrid additive–subtractive features across complexity levels
- Each CAD model is stored in STEP format and includes labeled B-Rep faces for hybrid manufacturing feature recognition
- The CAD models were generated programmatically using PythonOCC, ensuring consistent quality and scalability
- B-Rep hierarchical graph representations for GCNN training
Foundation Model Driven Robotics: A Comprehensive Review
Publication: Khan, M.T., Waheed, A., 2025. Foundation Model Driven Robotics: A Comprehensive Review. arXiv preprint arXiv:2507.10087. Under Review.
Overview: This comprehensive review examines the transformative impact of foundation models on robotics, analyzing how large language models (LLMs), vision-language models (VLMs), and multimodal architectures are revolutionizing robotic perception, decision-making, planning, and control. The paper provides a systematic analysis of state-of-the-art approaches, datasets, benchmarks, and future research directions in foundation model-driven robotics.
Scope: The review covers multiple aspects of foundation models in robotics:
- Language Models for Robotics: LLM-based task planning, code generation, reasoning, and human-robot interaction
- Vision-Language Models: Multimodal perception for manipulation, navigation, and grounded instruction following
- Multimodal Architectures: End-to-end learning frameworks integrating vision, language, and action spaces
- Simulation and Sim-to-Real Transfer: LLM-driven simulation environments and reality gap bridging techniques
Key Contributions
- Systematic taxonomy of foundation model architectures for robotics applications
- Analysis of LLM-driven frameworks for high-level task planning, code generation, and reasoning
- Evaluation of vision-language models for robotic manipulation and navigation tasks
- Overview of multimodal learning approaches combining perception, language understanding, and action prediction
- Identification of open challenges and future research directions in foundation model-driven robotics
Research Directions: The paper identifies critical challenges including sim-to-real transfer, real-time inference, safety and robustness, embodied intelligence, and scalability of foundation models for real-world robotic deployments.
Summary
This comprehensive research portfolio showcases the transformative application of foundation models (LLMs, VLMs, and multimodal architectures) across manufacturing intelligence and robotics. The work spans from automated engineering document understanding to intelligent robotic systems, demonstrating how AI-driven approaches are revolutionizing traditional manufacturing and automation workflows.
Manufacturing Intelligence & Automation
The manufacturing-focused projects leverage state-of-the-art vision-language models and deep learning to automate critical design and production tasks:
- Engineering Drawing Parsing: Hybrid frameworks combining YOLOv11-obb detection with OCR-free transformers (Donut, Florence-2) achieve >94% precision in extracting GD&T symbols, dimensions, tolerances, and manufacturing annotations from 2D technical drawings
- CAD Feature Recognition: Novel approaches using VLMs (GPT-4o, Claude-3.5-Sonnet) and graph neural networks enable automated manufacturability assessment and dimensional extraction from 3D CAD models
- Knowledge-Driven Decision Support: LLM-powered systems integrate structured knowledge graphs with natural language interfaces for material selection, process parameter optimization, and additive manufacturing planning
- Multi-Agent Process Planning: Ongoing development of generative AI agents for end-to-end manufacturing decision-making, from cost estimation to process planning
Foundation Models for Robotics
The comprehensive review of foundation model-driven robotics examines how LLMs and VLMs are enabling intelligent robotic systems capable of high-level reasoning, multimodal perception, and adaptive behavior in complex real-world environments.
Impact & Results
Across all projects, the research demonstrates:
- High Precision: >89% F1-scores and <11% hallucination rates in structured information extraction
- Practical Deployment: Real-world industrial case studies and interactive GUI applications
- Scalability: Large-scale datasets (161K+ CAD models, comprehensive engineering drawing corpora)
- Open Research: Published datasets, benchmarks, and reproducible methodologies advancing the field
These contributions establish a foundation for intelligent, automated manufacturing systems that significantly reduce manual effort, improve consistency, and enable data-driven decision-making throughout the product lifecycle.