diff --git a/starter_ai_agents/agentic_rag_math_agent/README.md b/rag_tutorials/agentic_rag_math_agent/README.md similarity index 87% rename from starter_ai_agents/agentic_rag_math_agent/README.md rename to rag_tutorials/agentic_rag_math_agent/README.md index 323a22c..3f8e8ef 100644 --- a/starter_ai_agents/agentic_rag_math_agent/README.md +++ b/rag_tutorials/agentic_rag_math_agent/README.md @@ -2,24 +2,19 @@ This project implements an **Agentic-RAG architecture** to simulate a math professor that solves **JEE-level math questions** with step-by-step explanations. The system smartly routes queries between a vector database and web search, applies input/output guardrails, and incorporates human feedback for continuous learning. ---- - ## 📌 Features - ✅ **Input Guardrails** (DSPy): Accepts only academic math questions. - 📚 **Knowledge Base Search**: Uses **Qdrant Vector DB** with OpenAI Embeddings to match known questions. - 🌐 **Web Fallback**: Integrates **Tavily API** when no good match is found. -- ✍️ **GPT-3.5 Turbo Explanations**: Generates step-by-step math solutions. +- ✍️ **GPT-4.1 Explanations**: Generates step-by-step math solutions. - 🛡️ **Output Guardrails**: Filters for correctness and safety. - 👍 **Human-in-the-Loop Feedback**: Users rate answers (Yes/No), logged for future learning. - 📊 **Benchmarking**: Evaluated on **JEEBench** dataset with adjustable question limits. - 💻 **Streamlit UI**: Interactive dashboard with multiple tabs. ---- - ## 🚀 Architecture Flow -![image](https://github.com/user-attachments/assets/9197a918-d14e-4759-9b28-8a90dadd1baf) ## 📚 Knowledge Base @@ -27,36 +22,29 @@ This project implements an **Agentic-RAG architecture** to simulate a math profe - **Vector DB:** Qdrant (with OpenAI Embeddings) - **Storage:** Built with `llama-index` to persist embeddings and perform top-1 similarity search ---- - ## 🌐 Web Search - Uses **Tavily API** for fallback search when the KB doesn't contain a good match -- Fetched content is piped into **GPT-3.5 Turbo** for clean explanation +- Fetched content is piped into **GPT-4o** for clean explanation ---- ## 🔐 Guardrails - **Input Guardrail (DSPy):** Accepts only math-related academic questions - **Output Guardrail (DSPy):** Blocks hallucinated or off-topic content ---- ## 👨‍🏫 Human-in-the-Loop Feedback - Streamlit UI allows students to give 👍 / 👎 after seeing the answer - Feedback is logged to a local JSON file for future improvement ---- - ## 📊 Benchmarking - Evaluated on **50 random JEEBench Math Questions** - **Current Accuracy:** 66% - Benchmark results saved to: `benchmark/results.csv` ---- ## 🚀 Demo @@ -65,8 +53,6 @@ To run the app with Streamlit: ```bash streamlit run app/streamlit.py ----- - diff --git a/starter_ai_agents/agentic_rag_math_agent/app/benchmark.py b/rag_tutorials/agentic_rag_math_agent/app/benchmark.py similarity index 100% rename from starter_ai_agents/agentic_rag_math_agent/app/benchmark.py rename to rag_tutorials/agentic_rag_math_agent/app/benchmark.py diff --git a/starter_ai_agents/agentic_rag_math_agent/app/streamlit.py b/rag_tutorials/agentic_rag_math_agent/app/streamlit.py similarity index 100% rename from starter_ai_agents/agentic_rag_math_agent/app/streamlit.py rename to rag_tutorials/agentic_rag_math_agent/app/streamlit.py diff --git a/starter_ai_agents/agentic_rag_math_agent/benchmark/results_math_5.csv b/rag_tutorials/agentic_rag_math_agent/benchmark/results_math_5.csv similarity index 100% rename from starter_ai_agents/agentic_rag_math_agent/benchmark/results_math_5.csv rename to rag_tutorials/agentic_rag_math_agent/benchmark/results_math_5.csv diff --git a/starter_ai_agents/agentic_rag_math_agent/benchmark/results_math_50.csv b/rag_tutorials/agentic_rag_math_agent/benchmark/results_math_50.csv similarity index 100% rename from starter_ai_agents/agentic_rag_math_agent/benchmark/results_math_50.csv rename to rag_tutorials/agentic_rag_math_agent/benchmark/results_math_50.csv diff --git a/starter_ai_agents/agentic_rag_math_agent/config/.env b/rag_tutorials/agentic_rag_math_agent/config/.env similarity index 100% rename from starter_ai_agents/agentic_rag_math_agent/config/.env rename to rag_tutorials/agentic_rag_math_agent/config/.env diff --git a/starter_ai_agents/agentic_rag_math_agent/data/load_gsm8k_data.py b/rag_tutorials/agentic_rag_math_agent/data/load_gsm8k_data.py similarity index 100% rename from starter_ai_agents/agentic_rag_math_agent/data/load_gsm8k_data.py rename to rag_tutorials/agentic_rag_math_agent/data/load_gsm8k_data.py diff --git a/starter_ai_agents/agentic_rag_math_agent/logs/feedback_log.json b/rag_tutorials/agentic_rag_math_agent/logs/feedback_log.json similarity index 83% rename from starter_ai_agents/agentic_rag_math_agent/logs/feedback_log.json rename to rag_tutorials/agentic_rag_math_agent/logs/feedback_log.json index 98cbd9f..dfd01a6 100644 --- a/starter_ai_agents/agentic_rag_math_agent/logs/feedback_log.json +++ b/rag_tutorials/agentic_rag_math_agent/logs/feedback_log.json @@ -63,5 +63,10 @@ "question": "Let \ud835\udc53 ( \ud835\udc65 ) = \ud835\udc65 3 \u2212 3 \ud835\udc65 + 1 f(x)=x 3 \u22123x+1. The number of real solutions of the equation \ud835\udc53 ( \ud835\udc53 ( \ud835\udc65 ) ) = 0 f(f(x))=0 is: Options: (A) 3 (B) 5 (C) 7 (D) 9", "answer": "To find the number of real solutions of the equation \ud835\udc53(\ud835\udc53(\ud835\udc65)) = 0, we first need to understand what the function \ud835\udc53(\ud835\udc65) = \ud835\udc65\u00b3 - 3\ud835\udc65 + 1 represents.\n\nThe function \ud835\udc53(\ud835\udc65) = \ud835\udc65\u00b3 - 3\ud835\udc65 + 1 is a cubic function, which means it has at most 3 real roots. To find the real roots of \ud835\udc53(\ud835\udc65), we set \ud835\udc53(\ud835\udc65) = 0 and solve for \ud835\udc65:\n\n\ud835\udc65\u00b3 - 3\ud835\udc65 + 1 = 0\n\nThis equation may have 1, 2, or 3 real roots. Let's assume it has 3 real roots for now.\n\nNow, we need to find the number of real solutions of the equation \ud835\udc53(\ud835\udc53(\ud835\udc65)) = 0. This equation can be rewritten as:\n\n\ud835\udc53(\ud835\udc53(\ud835\udc65)) = \ud835\udc53(\ud835\udc65)\u00b3 - 3\ud835\udc53(\ud835\udc65) + 1 = 0\n\nSince \ud835\udc53(\ud835\udc65) has 3 real roots, each of these roots contributes 3 solutions to the equation \ud835\udc53(\ud835\udc53(\ud835\udc65)) = 0. Therefore, the total number of real solutions of \ud835\udc53(\ud835\udc53(\ud835\udc65)) = 0 is 3 * 3 = 9.\n\nTherefore, the correct answer is (D) 9, as stated in the web content provided.", "feedback": "positive" + }, + { + "question": "Let \ud835\udc53 ( \ud835\udc65 ) = ln \u2061 ( sin \u2061 \ud835\udc65 ) + ln \u2061 ( tan \u2061 \ud835\udc65 ) f(x)=ln(sinx)+ln(tanx). Find the domain of the function \ud835\udc53 ( \ud835\udc65 ) f(x) in the interval ( 0 , \ud835\udf0b ) (0,\u03c0).", + "answer": "To find the domain of the function \\( f(x) = \\ln(\\sin x) + \\ln(\\tan x) \\) in the interval \\( (0, \\pi) \\), we need to determine where both \\(\\ln(\\sin x)\\) and \\(\\ln(\\tan x)\\) are defined.\n\n### Step-by-Step Explanation:\n\n1. **Understanding the Natural Logarithm:**\n - The natural logarithm function \\(\\ln(y)\\) is defined only for \\(y > 0\\).\n\n2. **Analyzing \\(\\ln(\\sin x)\\):**\n - \\(\\sin x > 0\\) in the interval \\( (0, \\pi) \\).\n - Therefore, \\(\\ln(\\sin x)\\) is defined for all \\(x\\) in \\( (0, \\pi) \\).\n\n3. **Analyzing \\(\\ln(\\tan x)\\):**\n - \\(\\tan x = \\frac{\\sin x}{\\cos x}\\).\n - For \\(\\ln(\\tan x)\\) to be defined, \\(\\tan x > 0\\), which means \\(\\sin x > 0\\) and \\(\\cos x > 0\\).\n - \\(\\cos x > 0\\) in the interval \\( (0, \\frac{\\pi}{2}) \\).\n - \\(\\tan x\\) is undefined at \\(x = \\frac{\\pi}{2}\\) because \\(\\cos x = 0\\) there.\n\n4. **Combining the Conditions:**\n - Both \\(\\ln(\\sin x)\\) and \\(\\ln(\\tan x)\\) are defined when \\(x\\) is in \\( (0, \\frac{\\pi}{2}) \\) and \\( (\\frac{\\pi}{2}, \\pi) \\).\n - At \\(x = \\frac{\\pi}{2}\\), \\(\\tan x\\) is undefined, so \\(x = \\frac{\\pi}{2}\\) is excluded from the domain.\n\n5. **Conclusion:**\n - The domain of \\( f(x) = \\ln(\\sin x) + \\ln(\\tan x) \\) in the interval \\( (0, \\pi) \\) is \\( (0, \\frac{\\pi}{2}) \\cup (\\frac{\\pi}{2}, \\pi) \\).\n\nThis step-by-step analysis confirms that the domain of the function \\( f(x) \\) is indeed \\( (0, \\frac{\\pi}{2}) \\cup (\\frac{\\pi}{2}, \\pi) \\), excluding \\(x = \\frac{\\pi}{2}\\) where \\(\\tan x\\) is undefined.", + "feedback": "positive" } ] \ No newline at end of file diff --git a/starter_ai_agents/agentic_rag_math_agent/rag/guardrails.py b/rag_tutorials/agentic_rag_math_agent/rag/guardrails.py similarity index 98% rename from starter_ai_agents/agentic_rag_math_agent/rag/guardrails.py rename to rag_tutorials/agentic_rag_math_agent/rag/guardrails.py index e42557c..ba6948a 100644 --- a/starter_ai_agents/agentic_rag_math_agent/rag/guardrails.py +++ b/rag_tutorials/agentic_rag_math_agent/rag/guardrails.py @@ -8,7 +8,7 @@ OPENAI_API_KEY = os.getenv("OPENAI_API_KEY") print("🔐 Loaded OPENAI_API_KEY:", "✅ Found" if OPENAI_API_KEY else "❌ Missing") # Configure LM -lm = dspy.LM(model="gpt-3.5-turbo", api_key=OPENAI_API_KEY) +lm = dspy.LM(model="gpt-4o", api_key=OPENAI_API_KEY) dspy.configure(lm=lm) # ✅ Signature for Input Guard diff --git a/starter_ai_agents/agentic_rag_math_agent/rag/query_router.py b/rag_tutorials/agentic_rag_math_agent/rag/query_router.py similarity index 96% rename from starter_ai_agents/agentic_rag_math_agent/rag/query_router.py rename to rag_tutorials/agentic_rag_math_agent/rag/query_router.py index d15039d..f66796e 100644 --- a/starter_ai_agents/agentic_rag_math_agent/rag/query_router.py +++ b/rag_tutorials/agentic_rag_math_agent/rag/query_router.py @@ -6,7 +6,7 @@ sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) import os import requests -import openai # ✅ now using the real OpenAI SDK +import openai import json import inspect from llama_index.core import StorageContext,load_index_from_storage @@ -78,7 +78,7 @@ Web Content: Now write a clear, accurate, and step-by-step explanation of the student's question. Only include valid math steps — do not guess or make up answers. """ - llm = OpenAI(api_key=OPENAI_API_KEY, model="gpt-3.5-turbo") + llm = OpenAI(api_key=OPENAI_API_KEY, model="gpt-4o") response = llm.complete(prompt) return response.text @@ -118,7 +118,7 @@ Do not change the final answer. You are only allowed to explain what is already Use the KB content as your only source. Do not guess or recalculate. """ - llm = OpenAI(api_key=OPENAI_API_KEY, model="gpt-3.5-turbo") + llm = OpenAI(api_key=OPENAI_API_KEY, model="gpt-4o") answer = llm.complete(prompt).text from_kb = True else: diff --git a/starter_ai_agents/agentic_rag_math_agent/rag/vector.py b/rag_tutorials/agentic_rag_math_agent/rag/vector.py similarity index 100% rename from starter_ai_agents/agentic_rag_math_agent/rag/vector.py rename to rag_tutorials/agentic_rag_math_agent/rag/vector.py diff --git a/starter_ai_agents/agentic_rag_math_agent/requirements.txt b/rag_tutorials/agentic_rag_math_agent/requirements.txt similarity index 100% rename from starter_ai_agents/agentic_rag_math_agent/requirements.txt rename to rag_tutorials/agentic_rag_math_agent/requirements.txt diff --git a/starter_ai_agents/agentic_rag_math_agent/app/__pycache__/benchmark.cpython-311.pyc b/starter_ai_agents/agentic_rag_math_agent/app/__pycache__/benchmark.cpython-311.pyc deleted file mode 100644 index 5a1156a..0000000 Binary files a/starter_ai_agents/agentic_rag_math_agent/app/__pycache__/benchmark.cpython-311.pyc and /dev/null differ diff --git a/starter_ai_agents/agentic_rag_math_agent/app/__pycache__/streamlit.cpython-311.pyc b/starter_ai_agents/agentic_rag_math_agent/app/__pycache__/streamlit.cpython-311.pyc deleted file mode 100644 index 9eebe5c..0000000 Binary files a/starter_ai_agents/agentic_rag_math_agent/app/__pycache__/streamlit.cpython-311.pyc and /dev/null differ diff --git a/starter_ai_agents/agentic_rag_math_agent/data/__pycache__/load_gsm8k_data.cpython-311.pyc b/starter_ai_agents/agentic_rag_math_agent/data/__pycache__/load_gsm8k_data.cpython-311.pyc deleted file mode 100644 index 21e10bf..0000000 Binary files a/starter_ai_agents/agentic_rag_math_agent/data/__pycache__/load_gsm8k_data.cpython-311.pyc and /dev/null differ diff --git a/starter_ai_agents/agentic_rag_math_agent/rag/__pycache__/dspy_guardrails.cpython-311.pyc b/starter_ai_agents/agentic_rag_math_agent/rag/__pycache__/dspy_guardrails.cpython-311.pyc deleted file mode 100644 index b5c21ff..0000000 Binary files a/starter_ai_agents/agentic_rag_math_agent/rag/__pycache__/dspy_guardrails.cpython-311.pyc and /dev/null differ diff --git a/starter_ai_agents/agentic_rag_math_agent/rag/__pycache__/guardrails.cpython-311.pyc b/starter_ai_agents/agentic_rag_math_agent/rag/__pycache__/guardrails.cpython-311.pyc deleted file mode 100644 index cd16e99..0000000 Binary files a/starter_ai_agents/agentic_rag_math_agent/rag/__pycache__/guardrails.cpython-311.pyc and /dev/null differ diff --git a/starter_ai_agents/agentic_rag_math_agent/rag/__pycache__/query_router.cpython-311.pyc b/starter_ai_agents/agentic_rag_math_agent/rag/__pycache__/query_router.cpython-311.pyc deleted file mode 100644 index c7a9322..0000000 Binary files a/starter_ai_agents/agentic_rag_math_agent/rag/__pycache__/query_router.cpython-311.pyc and /dev/null differ diff --git a/starter_ai_agents/agentic_rag_math_agent/storage/default__vector_store.json b/starter_ai_agents/agentic_rag_math_agent/storage/default__vector_store.json deleted file mode 100644 index c4f2a4d..0000000 Binary files a/starter_ai_agents/agentic_rag_math_agent/storage/default__vector_store.json and /dev/null differ diff --git a/starter_ai_agents/agentic_rag_math_agent/storage/docstore.json b/starter_ai_agents/agentic_rag_math_agent/storage/docstore.json deleted file mode 100644 index 9e26dfe..0000000 --- a/starter_ai_agents/agentic_rag_math_agent/storage/docstore.json +++ /dev/null @@ -1 +0,0 @@ -{} \ No newline at end of file diff --git a/starter_ai_agents/agentic_rag_math_agent/storage/faiss.index b/starter_ai_agents/agentic_rag_math_agent/storage/faiss.index deleted file mode 100644 index c4f2a4d..0000000 Binary files a/starter_ai_agents/agentic_rag_math_agent/storage/faiss.index and /dev/null differ diff --git a/starter_ai_agents/agentic_rag_math_agent/storage/graph_store.json b/starter_ai_agents/agentic_rag_math_agent/storage/graph_store.json deleted file mode 100644 index 9aab8ea..0000000 --- a/starter_ai_agents/agentic_rag_math_agent/storage/graph_store.json +++ /dev/null @@ -1 +0,0 @@ -{"graph_dict": {}} \ No newline at end of file diff --git a/starter_ai_agents/agentic_rag_math_agent/storage/image__vector_store.json b/starter_ai_agents/agentic_rag_math_agent/storage/image__vector_store.json deleted file mode 100644 index 8534c56..0000000 --- a/starter_ai_agents/agentic_rag_math_agent/storage/image__vector_store.json +++ /dev/null @@ -1 +0,0 @@ -{"embedding_dict": {}, "text_id_to_ref_doc_id": {}, "metadata_dict": {}} \ No newline at end of file diff --git a/starter_ai_agents/agentic_rag_math_agent/storage/index_store.json b/starter_ai_agents/agentic_rag_math_agent/storage/index_store.json deleted file mode 100644 index fc00b7b..0000000 --- a/starter_ai_agents/agentic_rag_math_agent/storage/index_store.json +++ /dev/null @@ -1 +0,0 @@ -{"index_store/data": {"b8b2a959-cac2-4df4-808b-9edf8eda98ff": {"__type__": "vector_store", "__data__": "{\"index_id\": \"b8b2a959-cac2-4df4-808b-9edf8eda98ff\", \"summary\": null, \"nodes_dict\": {}, \"doc_id_dict\": {}, \"embeddings_dict\": {}}"}}} \ No newline at end of file