From 5311fda5fccaf332c7e17d1b2fa144aaa15cbb94 Mon Sep 17 00:00:00 2001 From: Francy Lisboa Date: Sat, 18 Oct 2025 11:31:10 -0300 Subject: [PATCH] Add Agent Creator skill - American English version MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Complete translation of agent-creator meta-skill to American English. This skill teaches Claude Code to autonomously create production-ready agents using a 5-phase protocol: - Phase 1: Discovery (API research and selection) - Phase 2: Design (analysis definition) - Phase 3: Architecture (modular structure) - Phase 4: Detection (keyword identification) - Phase 5: Implementation (complete code generation) Includes comprehensive documentation (~24,000 words), quality standards, and detailed phase guides. Translation: Portuguese ā†’ American English Structure: Identical to original Quality: High-fidelity translation maintaining technical accuracy šŸ¤– Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- .claude-plugin/marketplace.json | 22 + .gitignore | 29 + README.md | 846 +++++++++ SKILL.md | 2693 +++++++++++++++++++++++++++ references/phase1-discovery.md | 454 +++++ references/phase2-design.md | 244 +++ references/phase3-architecture.md | 351 ++++ references/phase4-detection.md | 465 +++++ references/phase5-implementation.md | 1011 ++++++++++ references/phase6-testing.md | 783 ++++++++ references/quality-standards.md | 937 ++++++++++ 11 files changed, 7835 insertions(+) create mode 100644 .claude-plugin/marketplace.json create mode 100644 .gitignore create mode 100644 README.md create mode 100644 SKILL.md create mode 100644 references/phase1-discovery.md create mode 100644 references/phase2-design.md create mode 100644 references/phase3-architecture.md create mode 100644 references/phase4-detection.md create mode 100644 references/phase5-implementation.md create mode 100644 references/phase6-testing.md create mode 100644 references/quality-standards.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json new file mode 100644 index 0000000..a6691fe --- /dev/null +++ b/.claude-plugin/marketplace.json @@ -0,0 +1,22 @@ +{ + "name": "agent-creator-en", + "owner": { + "name": "Agent Creator", + "email": "noreply@example.com" + }, + "metadata": { + "description": "Agent Creator - Meta-skill for autonomous agent generation in English", + "version": "1.0.0", + "created": "2025-10-18", + "language": "en-US" + }, + "plugins": [ + { + "name": "agent-creator-en-plugin", + "description": "This skill should be used when the user asks to create an agent, automate a repetitive workflow, or create a custom skill. Activates with phrases like every day, daily I have to, I need to repeat, create agent for, automate workflow, create skill for, need to automate, turn process into agent. Claude will use the 5-phase protocol to research APIs, define analyses, structure everything, implement functional code, and create a complete skill autonomously in a subdirectory.", + "source": "./", + "strict": false, + "skills": ["./"] + } + ] +} diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..7be8a9c --- /dev/null +++ b/.gitignore @@ -0,0 +1,29 @@ +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +*.egg-info/ +dist/ +build/ + +# IDE +.vscode/ +.idea/ +*.swp +*.swo +*~ + +# OS +.DS_Store +Thumbs.db + +# Logs +*.log + +# Virtual environments +venv/ +.venv/ +ENV/ +env/ diff --git a/README.md b/README.md new file mode 100644 index 0000000..9c322ba --- /dev/null +++ b/README.md @@ -0,0 +1,846 @@ +# Agent Creator - Meta-Skill for Claude Code + +**Meta-skill that teaches Claude Code to create complete agents with Claude Skills in a fully autonomous way!** + +You describe a repetitive workflow ā†’ Claude creates a complete production-ready agent in 60-90 minutes. + +--- + +## šŸŽÆ What It Is and What It Does + +### Problem It Solves + +Creating a Claude Code agent manually is: +- ā° **Time-consuming**: 20-30 hours of work +- šŸ§  **Complex**: Requires knowledge of APIs, Python, architecture +- šŸ” **Labor-intensive**: Research APIs, define analyses, structure, code, document + +### Solution: Agent-Creator + +**You do:** +``` +"Automate this workflow: every day I download crop data, +compare current year vs previous, takes 2 hours." +``` + +**Claude Code does:** +1. šŸ” Research available APIs ā†’ Decide the best one +2. šŸŽØ Define useful analyses ā†’ Prioritize by value +3. šŸ—ļø Structure project ā†’ Optimal architecture +4. šŸ’» Implement Python code ā†’ Functional, no TODOs +5. šŸ“ Write documentation ā†’ 12,000+ words +6. āš™ļø Create configs ā†’ Real values +7. šŸ“¦ Deliver agent ā†’ Production-ready in subdirectory + +**Result:** Complete agent in ~90 minutes! + +--- + +## šŸš€ Quick Installation + +### Prerequisites + +- Claude Code CLI installed +- Python 3.8+ (for agents that will be created) + +### Step 1: Install in Claude Code + +**In the Claude Code terminal**, run: + +```bash +/plugin marketplace add FrancyJGLisboa/agent-creator-skill +``` + +### Step 2: Verify Installation + +```bash +/plugin list +``` + +You should see: +``` +āœ“ agent-creator +``` + +### Done! šŸŽ‰ + +The meta-skill is installed and ready to use. + +--- + +## šŸ’” How to Use (Tutorial Guide) + +### Basic Usage - Describe Your Workflow + +**In Claude Code**, simply describe what you do repeatedly: + +``` +šŸ‘¤ You type: +"Automate this workflow: every day I download US crop data, +compare current year vs previous, rank states. Takes 2 hours." +``` + +**šŸ¤– Claude Code:** +``` +[Detects keyword: "automate"] +[Activates skill: agent-creator] +[Loads 5-phase protocol] + +šŸ” Phase 1: Researching US agriculture APIs... + āœ“ 5 options found + āœ“ Decided: USDA NASS QuickStats API + Reason: Free, data since 1866, complete coverage + +šŸŽØ Phase 2: Defining analyses... + āœ“ 15 typical questions identified + āœ“ 5 main analyses: YoY, Ranking, Trends, Reports + +šŸ—ļø Phase 3: Structuring project... + āœ“ 3 scripts + 3 utils planned + +šŸŽÆ Phase 4: Defining automatic detection... + āœ“ 50+ keywords identified + +āš™ļø Phase 5: Implementing code... + āœ“ SKILL.md (6,200 words) + āœ“ fetch_nass.py (280 lines) + āœ“ parse_nass.py (180 lines) + āœ“ analyze_nass.py (520 lines) + āœ“ utils/ (350 lines) + āœ“ references/ (5,000 words) + āœ“ .claude-plugin/marketplace.json āœ… + āœ“ README.md (800 words) + +āœ… AGENT CREATED: ./usda-agriculture-agent/ + +šŸš€ To install: +/plugin marketplace add ./usda-agriculture-agent +``` + +### What You Get + +A complete directory: + +``` +./usda-agriculture-agent/ +ā”œā”€ā”€ .claude-plugin/ +ā”‚ ā””ā”€ā”€ marketplace.json ā† For Claude Code installation +ā”œā”€ā”€ SKILL.md ā† Complete orchestration (6,000 words) +ā”œā”€ā”€ scripts/ +ā”‚ ā”œā”€ā”€ fetch_nass.py ā† API client (280 lines) +ā”‚ ā”œā”€ā”€ parse_nass.py ā† Parsing & validation (180 lines) +ā”‚ ā”œā”€ā”€ analyze_nass.py ā† Analyses (520 lines) +ā”‚ ā””ā”€ā”€ utils/ +ā”‚ ā”œā”€ā”€ cache_manager.py ā† Smart cache +ā”‚ ā”œā”€ā”€ rate_limiter.py ā† Rate limiting +ā”‚ ā””ā”€ā”€ validators.py ā† Validations +ā”œā”€ā”€ references/ +ā”‚ ā”œā”€ā”€ api-guide.md ā† How to use the API +ā”‚ ā”œā”€ā”€ analysis-methods.md ā† Detailed methodologies +ā”‚ ā””ā”€ā”€ troubleshooting.md ā† Problem solving +ā”œā”€ā”€ assets/ +ā”‚ ā”œā”€ā”€ config.json ā† Real configurations +ā”‚ ā””ā”€ā”€ metadata.json ā† Metadata +ā”œā”€ā”€ DECISIONS.md ā† Decision justifications +ā””ā”€ā”€ README.md ā† Usage instructions +``` + +**Total:** ~2,000 lines of code + ~12,000 words of documentation + +--- + +## šŸ”„ How It Works Internally (5 Phases) + +### PHASE 1: Discovery (API Research) + +**What it does:** +- Research available public APIs for the domain +- Uses `WebSearch` and `WebFetch` to find options +- Compares APIs by: coverage, cost, rate limits, quality +- **DECIDES** autonomously which to use + +**Example (Agriculture):** +```bash +WebSearch: "US agriculture API free historical data" +WebSearch: "USDA API documentation" +WebFetch: https://quickstats.nass.usda.gov/api + +ā†’ DECISION: USDA NASS QuickStats API + Justification: Free, data since 1866, all crops +``` + +--- + +### PHASE 2: Design (Analysis Definition) + +**What it does:** +- Brainstorm 10-15 typical user questions +- Group by analysis type +- **DEFINES** 4-6 priority analyses +- Specifies methodologies (formulas, interpretations) + +**Example:** +``` +Typical questions: +- "Corn production in 2023?" +- "Compare soybeans 2024 vs 2023" +- "Top 10 producing states" + +ā†’ ANALYSES DEFINED: + 1. YoY Comparison (year vs year) + 2. State Ranking (top producers) + 3. Trend Analysis (trends) + 4. Report Generation (reports) +``` + +--- + +### PHASE 3: Architecture (Structuring) + +**What it does:** +- Defines folder and file structure +- Specifies responsibilities of each script +- Plans cache strategy and performance + +**Example:** +``` +ā†’ STRUCTURE: + scripts/ + - fetch_nass.py (API requests) + - parse_nass.py (parsing) + - analyze_nass.py (analyses) + utils/ + - cache_manager.py + - rate_limiter.py +``` + +--- + +### PHASE 4: Detection (Automatic Activation) + +**What it does:** +- Lists domain keywords +- Determines when skill should activate +- Creates optimized description + +**Example:** +``` +ā†’ KEYWORDS: + Entities: USDA, NASS, agriculture + Commodities: corn, soybeans, wheat + Metrics: production, area, yield + Actions: compare, ranking, trend +``` + +--- + +### PHASE 5: Implementation (Complete Code) + +**What it does:** +- Creates **marketplace.json** (REQUIRED!) +- Implements functional Python code +- Writes SKILL.md (5000+ words) +- Creates references with useful content +- Generates real configs + +**Commands executed internally:** +```bash +mkdir -p agent-name/{scripts/utils,references,assets,.claude-plugin} +# Write: .claude-plugin/marketplace.json +# Write: SKILL.md +# Write: scripts/fetch_*.py +# Write: scripts/parse_*.py +# Write: scripts/analyze_*.py +# Write: utils/cache_manager.py +# Write: references/*.md +# Write: assets/*.json +# Write: README.md +``` + +--- + +## šŸ“– Practical Usage Examples + +### Example 1: Agriculture (USDA) + +**šŸ‘¤ Input in Claude Code:** +``` +"Automate: every day I download US crop data, +compare current year vs previous, rank states. +Takes 2 hours." +``` + +**šŸ¤– Claude executes (autonomously):** +``` +Phase 1: Research ā†’ USDA NASS API +Phase 2: Define ā†’ YoY, Ranking, Trends +Phase 3: Structure ā†’ 3 scripts + 2 utils +Phase 4: Keywords ā†’ USDA, corn, soybeans, production +Phase 5: Implement ā†’ 1,410 lines of code + +āœ… Agent created: ./usda-agriculture-agent/ +``` + +**šŸ’» How to install the created agent:** +```bash +# In Claude Code +/plugin marketplace add ./usda-agriculture-agent +``` + +**šŸŽÆ How to use:** +``` +"US corn production in 2023" +"Compare soybeans this year vs last year" +"Top 10 wheat producing states" +``` + +--- + +### Example 2: Finance (Stocks) + +**šŸ‘¤ Input:** +``` +"Need agent for stock analysis: +fetch prices, calculate RSI and MACD, +compare with benchmarks. Daily, 3h." +``` + +**šŸ¤– Claude creates:** +``` +Phase 1: Alpha Vantage API (or Yahoo Finance) +Phase 2: Price fetching, RSI, MACD, Benchmarking +Phase 3-5: Implement everything + +āœ… Created: ./stock-technical-analysis-agent/ +``` + +**šŸ’» Install:** +```bash +/plugin marketplace add ./stock-technical-analysis-agent +``` + +--- + +### Example 3: Climate (Real Case from Today!) + +**šŸ‘¤ Input:** +``` +"Create agent for climate analysis of Sorriso, Mato Grosso. +Need to analyze temperature and precipitation, historical trends." +``` + +**šŸ¤– Claude executed (5 autonomous phases):** + +``` +šŸ” PHASE 1: Climate API Research + Researched: INMET, Open-Meteo, OpenWeather, Visual Crossing + Compared: Coverage, cost, historical data + āœ“ DECIDED: Open-Meteo Historical Weather API + ā€¢ Free (no API key!) + ā€¢ Data since 1940 (84+ years) + ā€¢ ERA5 reanalysis (high quality) + ā€¢ REST API (easy integration) + +šŸŽØ PHASE 2: Analysis Design + Brainstorm: 15 typical questions + āœ“ DEFINED: 6 main analyses + 1. Basic time series + 2. Temporal comparison (YoY) + 3. Trend analysis (CAGR, regression) + 4. Anomaly detection (Z-score) + 5. Seasonal analysis (monthly climatology) + 6. Descriptive statistics + +šŸ—ļø PHASE 3: Architecture + āœ“ STRUCTURED: + scripts/ + - fetch_climate.py (320 lines) + - parse_climate.py (180 lines) + - analyze_climate.py (420 lines) + utils/ + - cache_manager.py (350 lines) + - validators.py (450 lines) + - statistics.py (350 lines) + āœ“ Cache: Historical permanent, current year 24h + āœ“ Validations: Ranges, physical consistency, continuity + +šŸŽÆ PHASE 4: Automatic Detection + āœ“ KEYWORDS: Sorriso, temperature, precipitation, rain, climate, + trend, historical, anomaly, compare + āœ“ DESCRIPTION: 200 words optimized + +āš™ļø PHASE 5: Complete Implementation + āœ“ .claude-plugin/marketplace.json + āœ“ SKILL.md (6,800 words) + āœ“ Python scripts (2,070 lines) + āœ“ References (1,500 words) + āœ“ Configs (config.json, metadata.json) + āœ“ README.md + DECISIONS.md + +āœ… AGENT CREATED: ./climate-analysis-sorriso-mt/ +``` + +**šŸ“Š Statistics:** +- **Code:** 2,070 lines of Python +- **Documentation:** 13,600 words +- **Files:** 16 main files +- **Time:** ~90 minutes of autonomous creation + +**šŸ’» Installation of created agent:** +```bash +# In terminal +cd climate-analysis-sorriso-mt +pip install -r requirements.txt + +# In Claude Code +/plugin marketplace add ./climate-analysis-sorriso-mt +``` + +**šŸŽÆ Using the created agent:** +``` +šŸ‘¤ "What's the average temperature in Sorriso over the last 10 years?" +šŸ¤– [Skill activates automatically] + [Fetches data from API] + [Analyzes and responds] + +šŸ‘¤ "Rain trend in Sorriso since 1990" +šŸ¤– [34-year trend analysis] + [Returns rate of change, significance, projection] +``` + +--- + +## šŸ”„ How It Works: The 5 Autonomous Phases + +### PHASE 1: DISCOVERY (API Research) + +**Objective:** DECIDE which API to use + +**Process:** +1. Identifies domain (agriculture? finance? climate?) +2. Research available public APIs +3. Compares options (coverage, cost, quality) +4. **DECIDES** with justification +5. Documents decision + +**Autonomy:** Claude decides without asking the user! + +**Example of internal commands:** +```bash +# Claude executes internally: +WebSearch: "US agriculture API free historical data" +WebFetch: https://quickstats.nass.usda.gov/api +# Compares: NASS vs ERS vs FAO +# ā†’ DECISION: NASS (best option) +``` + +--- + +### PHASE 2: DESIGN (Analysis Definition) + +**Objective:** DEFINE which analyses to implement + +**Process:** +1. Brainstorm typical questions (10-15) +2. Group by type (comparisons, rankings, trends) +3. **DEFINES** 4-6 priority analyses +4. Specifies methodologies (mathematical formulas) + +**Autonomy:** Claude prioritizes by value and frequency of use! + +--- + +### PHASE 3: ARCHITECTURE (Structuring) + +**Objective:** STRUCTURE the project optimally + +**Process:** +1. Defines folder structure +2. Specifies scripts and responsibilities +3. Plans cache strategy +4. Defines validations + +**Autonomy:** Claude chooses optimal architecture based on complexity! + +--- + +### PHASE 4: DETECTION (Automatic Activation) + +**Objective:** DETERMINE keywords for detection + +**Process:** +1. Lists domain entities +2. Lists typical actions +3. Determines keywords +4. Creates optimized description (150-250 words) + +**Result:** Skill activates automatically when user asks relevant question! + +--- + +### PHASE 5: IMPLEMENTATION (Complete Code) + +**Objective:** IMPLEMENT everything with REAL code + +**Process:** +```bash +1. mkdir -p agent-name/{scripts/utils,references,assets,.claude-plugin} +2. Write: .claude-plugin/marketplace.json ā† REQUIRED! +3. Write: SKILL.md (5000+ words) +4. Write: scripts/*.py (functional code) +5. Write: utils/*.py (cache, validators, etc) +6. Write: references/*.md (useful content) +7. Write: assets/*.json (real configs) +8. Write: README.md + DECISIONS.md +``` + +**Quality Standards:** +- āœ… Complete code (no `TODO` or `pass`) +- āœ… Detailed docstrings +- āœ… Robust error handling +- āœ… Type hints +- āœ… Comprehensive validations + +**Result:** Production-ready agent! + +--- + +## šŸ“ Step-by-Step Commands + +### 1. Create an Agent + +**In Claude Code:** +``` +šŸ‘¤ "Create an agent for [objective]" + +OR + +šŸ‘¤ "Automate this workflow: [description]" +``` + +### 2. Wait for Creation + +Claude executes the 5 phases autonomously (~60-90 min) + +### 3. Install Created Agent + +**In terminal:** +```bash +# Go to agent directory +cd ./created-agent-name/ + +# Install Python dependencies +pip install -r requirements.txt + +# If API key needed (follow instructions in README) +export API_KEY_VAR="your_key_here" +``` + +**In Claude Code:** +```bash +# Install skill +/plugin marketplace add ./created-agent-name + +# Verify installation +/plugin list +``` + +### 4. Use the Agent + +**In Claude Code:** +``` +šŸ‘¤ Ask questions related to the domain +šŸ¤– Skill activates automatically and responds +``` + +--- + +## šŸŽÆ ROI (Return on Investment) + +| Metric | Manual | With Agent-Creator | Savings | +|---------|--------|-------------------|----------| +| **Time** | 20-30 hours | 1.5 hours | **15-20x** | +| **Required knowledge** | APIs, Python, Architecture | None | **100%** | +| **Code written** | By you | By Claude | **100%** | +| **Quality** | Variable | Production-ready | High | + +**But the best part:** You do nothing, just describe the workflow! šŸŽ‰ + +--- + +## šŸ“š Complete Documentation + +This repository includes detailed guides in Portuguese: + +1. **[como-compartilhar-skills.md](./como-compartilhar-skills.md)** + - How to publish your skills + - GitHub, ZIP, Claude.ai + - Best practices + +2. **[guia-completo-claude-skills.md](./guia-completo-claude-skills.md)** + - Complete guide about Claude Skills + - Technical specifications + - Examples + +3. **[como_instalar_agente_creator.md](./como_instalar_agente_creator.md)** + - Detailed installation instructions + - Troubleshooting + +4. **[meta-prompt-autonomo-criacao-agentes.md](./meta-prompt-autonomo-criacao-agentes.md)** + - Meta-prompt for agent creation + - Universal annotated template + - Quality checklist + +5. **[scripts-vs-skills-guia-didatico.md](./scripts-vs-skills-guia-didatico.md)** + - Didactic comparison Scripts vs Skills + - When to use each approach + +6. **[agent-creator/README.md](./agent-creator/README.md)** + - Meta-skill documentation + - Technical details + +--- + +## šŸ’” Use Cases + +### Agriculture +``` +"Create agent for Brazilian crop analysis via CONAB" +ā†’ Agent with TXT parsing, YoY analyses, regional rankings +``` + +### Finance +``` +"Automate daily stock analysis: prices, RSI, MACD" +ā†’ Agent with technical indicators, alerts, comparisons +``` + +### Climate +``` +"Climate analysis of Sorriso-MT: temperature, rain, trends" +ā†’ Agent with data since 1940, 6 types of analyses +``` + +### Economy +``` +"Agent for World Bank economic indicators" +ā†’ Agent with GDP, inflation, country comparisons +``` + +**Any domain with API or structured data!** + +--- + +## šŸ› ļø Useful Commands + +### Check Installed Skills +```bash +# In Claude Code +/plugin list +``` + +### Install Agent-Creator +```bash +# In Claude Code +/plugin marketplace add FrancyJGLisboa/agent-creator-skill +``` + +### Create an Agent +```bash +# In Claude Code (natural language) +"Create an agent for [objective]" +"Automate workflow of [description]" +``` + +### Install Created Agent +```bash +# Terminal +cd ./created-agent/ +pip install -r requirements.txt + +# Claude Code +/plugin marketplace add ./created-agent +``` + +### Use Agent +```bash +# In Claude Code (natural language) +Ask questions related to the agent's domain +``` + +--- + +## āš™ļø Technical Requirements + +### To Use Agent-Creator +- Claude Code CLI installed +- Internet connection (for API research) + +### For Created Agents +- Python 3.8+ +- pip (to install dependencies) +- Specific dependencies (listed in requirements.txt of each agent) +- API key (if chosen API requires - instructions in agent's README) + +--- + +## šŸŽ“ Understanding the Output + +### Main Files Created + +**`.claude-plugin/marketplace.json`** +- Configuration for Claude Code installation +- **CRITICAL:** Without it, skill cannot be installed + +**`SKILL.md`** +- Complete skill orchestration +- Detailed workflows +- Analysis documentation +- ~5000-7000 words + +**`scripts/`** +- Functional Python code +- Separated by responsibility (fetch, parse, analyze) +- Reusable utils (cache, validators) +- ~1500-2000 lines total + +**`references/`** +- Technical guides (API docs, methodologies) +- Troubleshooting +- Domain knowledge +- ~5000 words + +**`README.md`** +- Installation instructions +- Usage examples +- Troubleshooting + +**`DECISIONS.md`** +- Justifications for all decisions +- Which API chosen and why +- Which analyses and why +- Trade-offs considered + +--- + +## ā­ Features + +- āœ… **Total Autonomy:** Claude decides everything +- āœ… **Production-Ready:** Functional code, no TODOs +- āœ… **Complete Documentation:** 10,000+ words +- āœ… **Smart Cache:** TTL based on data type +- āœ… **Robust Validations:** Guaranteed data quality +- āœ… **Error Handling:** Retry, fallbacks, clear messages +- āœ… **Marketplace.json:** Guaranteed Claude Code installation + +--- + +## šŸš§ Limitations + +**DO NOT use for:** +- āŒ Editing existing skills (edit directly) +- āŒ Debugging skills (debug directly) +- āŒ Asking questions about skills (ask directly) + +**USE ONLY for:** +- āœ… Creating new agents from scratch +- āœ… Automating repetitive workflows + +--- + +## šŸ¤ Contributing + +Contributions are welcome! + +1. Fork this repository +2. Create a branch (`git checkout -b feature/improvement`) +3. Commit your changes +4. Push to the branch +5. Open a Pull Request + +--- + +## šŸ“„ License + +Apache 2.0 (same license as Anthropic's official skills) + +Free to use, modify, and distribute. + +--- + +## šŸ™ Credits + +**Inspired by:** +- [Anthropic Agent Skills Spec](https://github.com/anthropics/skills) +- [skill-creator skill](https://github.com/anthropics/skills/tree/main/skill-creator) + +**Differentiator:** Total autonomy - Claude decides everything, not just executes instructions. + +--- + +## šŸ“Š Repository Statistics + +**Agent-Creator Meta-Skill:** +- 8 main files +- ~5,000 words in SKILL.md +- 6 detailed references +- 5-phase autonomous protocol + +**Documentation:** +- 5 complete guides in Portuguese +- ~150 KB of documentation +- Complete coverage of Claude Skills ecosystem + +--- + +## šŸŒŸ Examples of Agents Created with Agent-Creator + +**1. USDA Agriculture Agent** +- API: USDA NASS +- Analyses: YoY, Ranking, Trends +- Output: 1,410 lines of code + +**2. Climate Analysis Sorriso-MT** (created today!) +- API: Open-Meteo +- Analyses: 6 types (series, trends, anomalies, etc.) +- Output: 2,070 lines of code + +**All created autonomously by the meta-skill!** + +--- + +## šŸ“ž Support + +**Issues:** https://github.com/FrancyJGLisboa/agent-creator-skill/issues +**Discussions:** https://github.com/FrancyJGLisboa/agent-creator-skill/discussions + +--- + +## šŸš€ Quick Start + +```bash +# 1. Install agent-creator +/plugin marketplace add FrancyJGLisboa/agent-creator-skill + +# 2. Create an agent (in Claude Code) +"Automate workflow for analyzing [your domain]" + +# 3. Wait for creation (~60-90 min) + +# 4. Install created agent +/plugin marketplace add ./created-agent + +# 5. Use it! +"[Ask domain questions]" +``` + +--- + +**Start automating today! Transform repetitive workflows into powerful agents! šŸš€** + +--- + +**Version:** 1.0.0 +**Updated:** October 2025 +**Author:** Created with Claude Code +**Repository:** https://github.com/FrancyJGLisboa/agent-creator-skill diff --git a/SKILL.md b/SKILL.md new file mode 100644 index 0000000..52283e9 --- /dev/null +++ b/SKILL.md @@ -0,0 +1,2693 @@ +--- +name: agent-creator-en +description: This skill should be used when the user asks to create an agent, automate a repetitive workflow, or create a custom skill. Activates with phrases like every day, daily I have to, I need to repeat, create agent for, automate workflow, create skill for, need to automate, turn process into agent. Claude will use the 5-phase protocol to research APIs, define analyses, structure everything, implement functional code, and create a complete skill autonomously in a subdirectory. +--- +# Agent Creator - Meta-Skill + +This skill teaches Claude Code how to autonomously create complete agents with Claude Skills. + +## When to Use This Skill + +Claude should automatically activate this skill when the user: + +āœ… **Asks to create an agent** + +- "Create an agent for [objective]" +- "I need an agent that [description]" +- "Develop an agent to automate [workflow]" + +āœ… **Asks to automate a workflow** + +- "Automate this process: [description]" +- "Every day I do [repetitive task], automate this" +- "Turn this workflow into an agent" + +āœ… **Asks to create a skill** + +- "Create a skill for [objective]" +- "Develop a custom skill for [domain]" + +āœ… **Describes a repetitive process** + +- "Every day I [process]... takes Xh" +- "I repeatedly need to [task]" +- "Manual workflow: [description]" + +## Overview + +When activated, this skill guides Claude through **5 autonomous phases** to create a complete production-ready agent: + +``` +PHASE 1: DISCOVERY +ā”œā”€ Research available APIs +ā”œā”€ Compare options +ā””ā”€ DECIDE which to use (with justification) + +PHASE 2: DESIGN +ā”œā”€ Think about use cases +ā”œā”€ DEFINE useful analyses +ā””ā”€ Specify methodologies + +PHASE 3: ARCHITECTURE +ā”œā”€ STRUCTURE folders and files +ā”œā”€ Define necessary scripts +ā””ā”€ Plan caching and performance + +PHASE 4: DETECTION +ā”œā”€ DETERMINE keywords +ā””ā”€ Create precise description + +PHASE 5: IMPLEMENTATION +ā”œā”€ šŸšØ FIRST: Create marketplace.json (MANDATORY!) +ā”œā”€ Create SKILL.md (5000+ words) +ā”œā”€ Implement Python scripts (functional!) +ā”œā”€ Write references (useful!) +ā”œā”€ Generate configs (real!) +ā”œā”€ Create README +ā””ā”€ āœ… FINAL: Test installation +``` + +**Output**: Complete agent in subdirectory ready to install. + +## Autonomous Creation Protocol + +### Fundamental Principles + +**Autonomy**: + +- āœ… Claude DECIDES which API to use (doesn't ask user) +- āœ… Claude DEFINES which analyses to perform (based on value) +- āœ… Claude STRUCTURES optimally (best practices) +- āœ… Claude IMPLEMENTS complete code (no placeholders) + +**Quality**: + +- āœ… Production-ready code (no TODOs) +- āœ… Useful documentation (not "see docs") +- āœ… Real configs (no placeholders) +- āœ… Robust error handling + +**Completeness**: + +- āœ… Complete SKILL.md (5000+ words) +- āœ… Functional scripts (1000+ lines total) +- āœ… References with content (3000+ words) +- āœ… Valid assets/configs +- āœ… README with instructions + +### Requirements Extraction + +When user describes workflow vaguely, extract: + +**From what the user said**: + +- Domain (agriculture? finance? weather?) +- Data source (mentioned? if not, research) +- Main tasks (download? analyze? compare?) +- Frequency (daily? weekly? on-demand?) +- Current time spent (to calculate ROI) + +**Questions to ask** (only if critical and not inferable): + +- "Prefer free API or paid is ok?" +- "Need historical data for how many years?" +- "Focus on which geography/country?" + +**Rule**: Minimize questions. Infer/decide whenever possible. + +## PHASE 1: Discovery and Research + +**Objective**: DECIDE which API/data source to use + +### Process + +**1.1 Identify domain** + +From user input, what is the domain? + +- Agriculture ā†’ APIs: USDA NASS, FAO, World Bank Ag +- Finance ā†’ APIs: Alpha Vantage, Yahoo Finance, Fed Economic Data +- Weather ā†’ APIs: NOAA, OpenWeather, Weather.gov +- Economy ā†’ APIs: World Bank, IMF, FRED + +**1.2 Research available APIs** + +For the domain, use WebSearch to find: + +- Available public APIs +- Documentation +- Characteristics (free? rate limits? coverage?) + +**Example**: + +``` +WebSearch: "US agriculture API free historical data" +WebSearch: "USDA API documentation" +WebFetch: [doc URLs found] +``` + +**1.3 Compare options** + +Create mental table comparing: + +- Data coverage (fit with need) +- Cost (free vs paid) +- Rate limits (sufficient?) +- Data quality (official? reliable?) +- Documentation (good? examples?) +- Ease of use + +**1.4 DECIDE** + +Choose 1 API and justify. + +**Document decision** in separate file: + +```markdown +# Architecture Decisions + +## Selected API: [Name] + +**Justification**: +- āœ… Coverage: [details] +- āœ… Cost: [free/paid] +- āœ… Rate limit: [number] +- āœ… Quality: [official/private] +- āœ… Docs: [quality] + +**Alternatives considered**: +- API X: Rejected because [reason] +- API Y: Rejected because [reason] + +**Conclusion**: [Chosen API] is the best option because [synthesis] +``` + +**1.5 Research technical details** + +Use WebFetch to load API documentation and extract: + +- Base URL +- Main endpoints +- Authentication +- Important parameters +- Response format +- Rate limits +- Request/response examples + +**See** `references/phase1-discovery.md` for complete details. + +## PHASE 2: Analysis Design + +**Objective**: DEFINE which analyses the agent will perform + +### Process + +**2.1 Think about use cases** + +For the described workflow, which questions will the user ask frequently? + +**Brainstorm**: List 10-15 typical questions + +**2.2 Group by analysis type** + +Group similar questions: + +- Simple queries (fetch + format) +- Temporal comparisons (YoY) +- Rankings (sort + share) +- Trends (time series + CAGR) +- Projections (forecasting) +- Aggregations (regional/categorical) + +**2.3 DEFINE priority analyses** + +Choose 4-6 analyses that cover 80% of use cases. + +For each analysis: + +- Name +- Objective +- Required inputs +- Expected outputs +- Methodology (formulas, transformations) +- Interpretation + +**2.4 ADD Comprehensive Report Function** (šŸ†• Enhancement #8 - MANDATORY!) + +**āš ļø COMMON PROBLEM:** v1.0 skills had isolated functions. When user asks for "complete report", Claude didn't know how to combine all analyses. + +**Solution:** ALWAYS include as last analysis function: + +```python +def comprehensive_{domain}_report( + entity: str, + year: Optional[int] = None, + include_metrics: Optional[List[str]] = None, + client: Optional[Any] = None +) -> Dict: + """ + Generate comprehensive report combining ALL available metrics. + + This is a "one-stop" function that users can call to get + complete picture without knowing individual functions. + + Args: + entity: Entity to analyze (e.g., commodity, stock, location) + year: Year (None for current year with auto-detection) + include_metrics: Which metrics to include (None = all available) + client: API client instance (optional, created if None) + + Returns: + Dict with ALL metrics consolidated: + { + 'entity': str, + 'year': int, + 'year_info': str, + 'generated_at': str (ISO timestamp), + 'metrics': { + 'metric1_name': {metric1_data}, + 'metric2_name': {metric2_data}, + ... + }, + 'summary': str (overall insights), + 'alerts': List[str] (important findings) + } + + Example: + >>> report = comprehensive_{domain}_report("CORN") + >>> print(report['summary']) + "CORN 2025: Production up 5% YoY, yield at record high..." + """ + from datetime import datetime + from utils.helpers import get_{domain}_year_with_fallback, format_year_message + + # Auto-detect year + year_requested = year + if year is None: + year, _ = get_{domain}_year_with_fallback() + + # Initialize report + report = { + 'entity': entity, + 'year': year, + 'year_requested': year_requested, + 'year_info': format_year_message(year, year_requested), + 'generated_at': datetime.now().isoformat(), + 'metrics': {}, + 'alerts': [] + } + + # Determine which metrics to include + if include_metrics is None: + # Include ALL available metrics + metrics_to_fetch = ['{metric1}', '{metric2}', '{metric3}', ...] + else: + metrics_to_fetch = include_metrics + + # Call ALL individual analysis functions + # Graceful degradation: if one fails, others still run + + if '{metric1}' in metrics_to_fetch: + try: + report['metrics']['{metric1}'] = {metric1}_analysis(entity, year, client) + except Exception as e: + report['metrics']['{metric1}'] = { + 'error': str(e), + 'status': 'unavailable' + } + report['alerts'].append(f"{metric1} data unavailable: {e}") + + if '{metric2}' in metrics_to_fetch: + try: + report['metrics']['{metric2}'] = {metric2}_analysis(entity, year, client) + except Exception as e: + report['metrics']['{metric2}'] = { + 'error': str(e), + 'status': 'unavailable' + } + + # Repeat for ALL metrics... + + # Generate summary based on all available data + report['summary'] = _generate_summary(report['metrics'], entity, year) + + # Detect important findings + report['alerts'].extend(_detect_alerts(report['metrics'])) + + return report + + +def _generate_summary(metrics: Dict, entity: str, year: int) -> str: + """Generate human-readable summary from all metrics.""" + insights = [] + + # Extract key insights from each metric + for metric_name, metric_data in metrics.items(): + if 'error' not in metric_data: + # Extract most important insight from this metric + key_insight = _extract_key_insight(metric_name, metric_data) + if key_insight: + insights.append(key_insight) + + # Combine into coherent summary + if insights: + summary = f"{entity} {year}: " + ". ".join(insights[:3]) # Top 3 insights + else: + summary = f"{entity} {year}: No data available" + + return summary + + +def _detect_alerts(metrics: Dict) -> List[str]: + """Detect significant findings that need attention.""" + alerts = [] + + # Check each metric for alert conditions + for metric_name, metric_data in metrics.items(): + if 'error' in metric_data: + continue + + # Domain-specific alert logic + # Example: Large changes, extreme values, anomalies + if metric_name == '{metric1}' and 'change_percent' in metric_data: + if abs(metric_data['change_percent']) > 15: + alerts.append( + f"āš  Large {metric1} change: {metric_data['change_percent']:.1f}%" + ) + + return alerts +``` + +**Why it's mandatory:** +- āœ… Users want "complete report" ā†’ 1 function does everything +- āœ… Ideal for executive dashboards +- āœ… Facilitates sales ("everything in one report") +- āœ… Much better UX (no need to know individual functions) + +**When to mention in SKILL.md:** + +```markdown +## Comprehensive Analysis (All-in-One) + +To get a complete report combining ALL metrics: + +Use the `comprehensive_{domain}_report()` function. + +This function: +- Fetches ALL available metrics +- Combines into single report +- Generates automatic summary +- Detects important alerts +- Degrades gracefully (if 1 metric fails, others work) + +Usage example: +"Generate complete report for {entity}" +"Complete dashboard for {entity}" +"All metrics for {entity}" +``` + +**Impact:** +- āœ… 10x better UX (1 query = everything) +- āœ… More useful skills for end users +- āœ… Facilitates commercial adoption + +**2.5 Specify methodologies** + +For quantitative analyses, define: + +- Mathematical formulas +- Statistical validations +- Interpretations +- Edge cases + +**See** `references/phase2-design.md` for detailed methodologies. + +## PHASE 3: Architecture + +**Objective**: STRUCTURE the agent optimally + +### Process + +**3.1 Define folder structure** + +Based on analyses and API: + +``` +agent-name/ +ā”œā”€ā”€ SKILL.md +ā”œā”€ā”€ scripts/ +ā”‚ ā”œā”€ā”€ [fetch/parse/analyze separate or together?] +ā”‚ ā””ā”€ā”€ utils/ +ā”‚ ā””ā”€ā”€ [cache? rate limiter? validators?] +ā”œā”€ā”€ references/ +ā”‚ ā””ā”€ā”€ [API docs? methodologies? troubleshooting?] +ā””ā”€ā”€ assets/ + ā””ā”€ā”€ [configs? metadata?] +``` + +**Decisions**: + +- Separate scripts (modular) vs monolithic? +- Which utilities needed? +- Which references useful? +- Which configs/assets? + +**3.2 Define responsibilities** + +For each script, specify: + +- File name +- Function/purpose +- Input and output +- Specific responsibilities +- ~Expected number of lines + +**3.3 Plan references** + +Which reference files to create? + +- API guide (how to use API) +- Analysis methods (methodologies) +- Troubleshooting (common errors) +- Domain knowledge (domain context) + +**3.4 Performance strategy** + +- Cache: What to cache? TTL? +- Rate limiting: How to control? +- Optimizations: Parallelization? Lazy loading? + +**See** `references/phase3-architecture.md` for structuring patterns. + +## PHASE 4: Automatic Detection + +**Objective**: DETERMINE keywords for automatic activation + +### Process + +**4.1 List domain entities** + +- Organizations/data sources +- Main metrics +- Geography (countries, regions, states) +- Temporality (years, periods) + +**4.2 List typical actions** + +- Query: "what", "how much", "show" +- Compare: "compare", "vs", "versus" +- Rank: "top", "best", "ranking" +- Analyze: "trend", "growth", "analyze" +- Forecast: "predict", "project", "forecast" + +**4.3 List question variations** + +For each analysis type, how might the user ask? + +**4.4 Define negative scope** + +Important! What should NOT activate the skill? + +**4.5 Create precise description** + +With all keywords identified, create ~200 word description that: + +- Mentions domain +- Lists main keywords +- Gives examples +- Defines negative scope + +**See** `references/phase4-detection.md` for complete guide. + +## PHASE 5: Complete Implementation + +**Objective**: IMPLEMENT everything with REAL code + +### āš ļø MANDATORY QUALITY STANDARDS + +Before starting implementation, read `references/quality-standards.md`. + +**NEVER DO**: + +- āŒ `# TODO: implement` +- āŒ `pass` in functions +- āŒ "See external documentation" +- āŒ Configs with "YOUR_KEY_HERE" without instructions +- āŒ Empty references or just links + +**ALWAYS DO**: + +- āœ… Complete and functional code +- āœ… Detailed docstrings +- āœ… Robust error handling +- āœ… Type hints +- āœ… Validations +- āœ… Real content in references +- āœ… Configs with real values + +### šŸšØ STEP 0: BEFORE EVERYTHING - Marketplace.json (MANDATORY) + +**STOP! READ THIS BEFORE CONTINUING!** + +šŸ›‘ **CRITICAL BLOCKER**: You CANNOT create ANY other file until completing this step. + +**Why marketplace.json is step 0:** + +- āŒ Without this file, the skill CANNOT be installed via `/plugin marketplace add` +- āŒ All the work creating the agent will be USELESS without it +- āŒ This is the most common error when creating agents - DO NOT make this mistake! + +#### Step 0.1: Create basic structure + +```bash +mkdir -p agent-name/.claude-plugin +``` + +#### Step 0.2: Create marketplace.json IMMEDIATELY + +Create `.claude-plugin/marketplace.json` with this content: + +```json +{ + "name": "agent-name", + "owner": { + "name": "Agent Creator", + "email": "noreply@example.com" + }, + "metadata": { + "description": "Brief agent description", + "version": "1.0.0", + "created": "2025-10-17" + }, + "plugins": [ + { + "name": "agent-plugin", + "description": "THIS DESCRIPTION MUST BE IDENTICAL to the description in SKILL.md frontmatter that you'll create in the next step", + "source": "./", + "strict": false, + "skills": ["./"] + } + ] +} +``` + +**āš ļø CRITICAL FIELDS:** + +- `name`: Agent name (same as directory name) +- `plugins[0].description`: **MUST BE EXACTLY EQUAL** to SKILL.md frontmatter description +- `plugins[0].skills`: `["./"]` points to SKILL.md in root +- `plugins[0].source`: `"./"` points to agent root + +#### Step 0.3: VALIDATE IMMEDIATELY (before continuing!) + +**Execute NOW these validation commands:** + +```bash +# 1. Validate JSON syntax +python3 -c "import json; print('āœ… Valid JSON'); json.load(open('agent-name/.claude-plugin/marketplace.json'))" + +# 2. Verify file exists +ls -la agent-name/.claude-plugin/marketplace.json + +# If any command fails: STOP and fix before continuing! +``` + +**āœ… CHECKLIST - You MUST complete ALL before proceeding:** + +- [ ] āœ… File `.claude-plugin/marketplace.json` created +- [ ] āœ… JSON is syntactically valid (validated with python) +- [ ] āœ… Field `name` is correct +- [ ] āœ… Field `plugins[0].description` ready to receive SKILL.md description +- [ ] āœ… Field `plugins[0].skills` = `["./"]` +- [ ] āœ… Field `plugins[0].source` = `"./"` + +**šŸ›‘ ONLY PROCEED AFTER VALIDATING ALL ITEMS ABOVE!** + +--- + +### Implementation Order (AFTER marketplace.json validated) + +Now that marketplace.json is created and validated, proceed: + +**1. Create rest of directory structure** + +```bash +mkdir -p agent-name/{scripts/utils,references,assets,data/{raw,processed,cache,analysis}} +``` + +**2. Create SKILL.md** + +Mandatory structure: + +- Frontmatter (name, description) +- When to use +- How it works (overview) +- Data source (detailed API) +- Workflows (step-by-step by question type) +- Available scripts (each explained) +- Available analyses (each explained) +- Error handling (all expected errors) +- Mandatory validations +- Performance and cache +- Keywords for detection +- Usage examples (5+ complete) + +**Size**: 5000-7000 words + +**āš ļø AFTER creating SKILL.md: SYNCHRONIZE description with marketplace.json!** + +**CRITICAL**: Now that SKILL.md is created with its frontmatter, you MUST: + +```bash +# Edit marketplace.json to update description +# Copy EXACTLY the description from SKILL.md frontmatter +# Paste in .claude-plugin/marketplace.json ā†’ plugins[0].description +``` + +**Verify synchronization:** + +- SKILL.md frontmatter description = marketplace.json plugins[0].description +- Must be IDENTICAL (word for word!) +- Without this, skill won't activate automatically + +**3. Implement Python scripts** + +**Order** (MANDATORY): + +1. **Utils first** (including helpers.py + validators/ - CRITICAL!) + - `utils/helpers.py` (šŸ”“ MANDATORY - already specified previously) + - `utils/cache_manager.py` + - `utils/rate_limiter.py` + - `utils/validators/` (šŸ”“ MANDATORY - see Step 3.5 below) +2. **Fetch** (API client - 1 method per API metric) +3. **Parse** (šŸ”“ MODULAR: 1 parser per data type! - see Step 3.2 below) +4. **Analyze** (analyses - include comprehensive_report already specified!) + +**Each script (in general)**: + +- Shebang: `#!/usr/bin/env python3` +- Complete module docstring +- Organized imports +- Classes/functions with docstrings +- Type hints +- Error handling +- Logging +- Main function with argparse +- if __name__ == "__main__" + +--- + +### Step 3.2: Modular Parser Architecture (šŸ†• Enhancement #5 - MANDATORY!) + +**āš ļø COMMON PROBLEM:** v1.0 had 1 generic parser. When adding new data types, architecture broke. + +**Solution:** **1 specific parser per API data type!** + +**Rule:** If API returns N data types (identified in Phase 1.6) ā†’ create N specific parsers + +**Mandatory structure:** + +``` +scripts/ +ā”œā”€ā”€ parse_{type1}.py # Ex: parse_conditions.py +ā”œā”€ā”€ parse_{type2}.py # Ex: parse_progress.py +ā”œā”€ā”€ parse_{type3}.py # Ex: parse_yield.py +ā”œā”€ā”€ parse_{type4}.py # Ex: parse_production.py +ā””ā”€ā”€ parse_{type5}.py # Ex: parse_area.py +``` + +**Template for each parser:** + +```python +#!/usr/bin/env python3 +""" +Parser for {type} data from {API_name}. +Handles {type}-specific transformations and validations. +""" + +import pandas as pd +from typing import List, Dict, Any, Optional +import logging + +logger = logging.getLogger(__name__) + + +def parse_{type}_response(data: List[Dict]) -> pd.DataFrame: + """ + Parse API response for {type} data. + + Args: + data: Raw API response (list of dicts) + + Returns: + DataFrame with standardized schema: + - entity: str + - year: int + - {type}_value: float + - unit: str + - {type}_specific_fields: various + + Raises: + ValueError: If data is invalid + ParseError: If parsing fails + + Example: + >>> data = [{'entity': 'CORN', 'year': 2025, 'value': '15,300,000'}] + >>> df = parse_{type}_response(data) + >>> df.shape + (1, 5) + """ + if not data: + raise ValueError("Data cannot be empty") + + # Convert to DataFrame + df = pd.DataFrame(data) + + # {Type}-specific transformations + df = _clean_{type}_values(df) + df = _extract_{type}_metadata(df) + df = _standardize_{type}_schema(df) + + # Validate + _validate_{type}_schema(df) + + return df + + +def _clean_{type}_values(df: pd.DataFrame) -> pd.DataFrame: + """Clean {type}-specific values (remove formatting, convert types).""" + # Example: Remove commas from numbers + if 'value' in df.columns: + df['value'] = df['value'].astype(str).str.replace(',', '') + df['value'] = pd.to_numeric(df['value'], errors='coerce') + + # {Type}-specific cleaning + # ... + + return df + + +def _extract_{type}_metadata(df: pd.DataFrame) -> pd.DataFrame: + """Extract {type}-specific metadata fields.""" + # Example for progress data: extract % from "75% PLANTED" + # Example for condition data: extract rating from "GOOD (60%)" + # Customize per data type! + + return df + + +def _standardize_{type}_schema(df: pd.DataFrame) -> pd.DataFrame: + """ + Standardize column names and schema for {type} data. + + Output schema: + - entity: str + - year: int + - {type}_value: float (main metric) + - unit: str + - additional_{type}_fields: various + """ + # Rename columns to standard names + column_mapping = { + 'api_entity_field': 'entity', + 'api_year_field': 'year', + 'api_value_field': '{type}_value', + # Add more as needed + } + df = df.rename(columns=column_mapping) + + # Ensure types + df['year'] = df['year'].astype(int) + df['{type}_value'] = pd.to_numeric(df['{type}_value'], errors='coerce') + + return df + + +def _validate_{type}_schema(df: pd.DataFrame) -> None: + """Validate {type} DataFrame schema.""" + required_columns = ['entity', 'year', '{type}_value'] + + missing = set(required_columns) - set(df.columns) + if missing: + raise ValueError(f"Missing required columns: {missing}") + + # Type validations + if not pd.api.types.is_integer_dtype(df['year']): + raise TypeError("'year' must be integer type") + + if not pd.api.types.is_numeric_dtype(df['{type}_value']): + raise TypeError("'{type}_value' must be numeric type") + + +def aggregate_{type}(df: pd.DataFrame, by: str) -> pd.DataFrame: + """ + Aggregate {type} data by specified level. + + Args: + df: Parsed {type} DataFrame + by: Aggregation level ('national', 'state', 'region') + + Returns: + Aggregated DataFrame + + Example: + >>> agg = aggregate_{type}(df, by='state') + """ + # Aggregation logic specific to {type} + if by == 'national': + return df.groupby(['year']).agg({ + '{type}_value': 'sum', + # Add more as needed + }).reset_index() + + elif by == 'state': + return df.groupby(['year', 'state']).agg({ + '{type}_value': 'sum', + }).reset_index() + + # Add more levels... + + +def format_{type}_report(df: pd.DataFrame) -> str: + """ + Format {type} data as human-readable report. + + Args: + df: Parsed {type} DataFrame + + Returns: + Formatted string report + + Example: + >>> report = format_{type}_report(df) + >>> print(report) + "{Type} Report: ..." + """ + lines = [f"## {Type} Report\n"] + + # Format based on {type} data + # Customize per type! + + return "\n".join(lines) + + +def main(): + """Test parser with sample data.""" + # Sample data for testing + sample_data = [ + { + 'entity': 'CORN', + 'year': 2025, + 'value': '15,300,000', + # Add {type}-specific fields + } + ] + + print("Testing parse_{type}_response()...") + df = parse_{type}_response(sample_data) + print(f"āœ“ Parsed {len(df)} records") + print(f"āœ“ Columns: {list(df.columns)}") + print(f"\n{df.head()}") + + print("\nTesting aggregate_{type}()...") + agg = aggregate_{type}(df, by='national') + print(f"āœ“ Aggregated: {agg}") + + print("\nTesting format_{type}_report()...") + report = format_{type}_report(df) + print(report) + + +if __name__ == "__main__": + main() +``` + +**Why create modular parsers:** +- āœ… Each data type has peculiarities (progress has %, yield has bu/acre, etc) +- āœ… Scalable architecture (easy to add new types) +- āœ… Isolated tests (each parser tested independently) +- āœ… Simple maintenance (bug in 1 type doesn't affect others) +- āœ… Organized code (clear responsibilities) + +**Impact:** Professional and scalable architecture from v1.0! + +--- + +### Step 3.5: Validation System (šŸ†• Enhancement #10 - MANDATORY!) + +**āš ļø COMMON PROBLEM:** v1.0 without data validation. User doesn't know if data is reliable. + +**Solution:** Complete validation system in `utils/validators/` + +**Mandatory structure:** + +``` +scripts/utils/validators/ +ā”œā”€ā”€ __init__.py +ā”œā”€ā”€ parameter_validator.py # Validate function parameters +ā”œā”€ā”€ data_validator.py # Validate API responses +ā”œā”€ā”€ temporal_validator.py # Validate temporal consistency +ā””ā”€ā”€ completeness_validator.py # Validate data completeness +``` + +**Template 1: parameter_validator.py** + +```python +#!/usr/bin/env python3 +""" +Parameter validators for {skill-name}. +Validates user inputs before making API calls. +""" + +from typing import Any, List, Optional +from datetime import datetime + + +class ValidationError(Exception): + """Raised when validation fails.""" + pass + + +def validate_entity(entity: str, valid_entities: Optional[List[str]] = None) -> str: + """ + Validate entity parameter. + + Args: + entity: Entity name (e.g., "CORN", "SOYBEANS") + valid_entities: List of valid entities (None to skip check) + + Returns: + str: Validated and normalized entity name + + Raises: + ValidationError: If entity is invalid + + Example: + >>> validate_entity("corn") + "CORN" # Normalized to uppercase + """ + if not entity: + raise ValidationError("Entity cannot be empty") + + if not isinstance(entity, str): + raise ValidationError(f"Entity must be string, got {type(entity)}") + + # Normalize + entity = entity.strip().upper() + + # Check if valid (if list provided) + if valid_entities and entity not in valid_entities: + suggestions = [e for e in valid_entities if entity[:3] in e] + raise ValidationError( + f"Invalid entity: {entity}\n" + f"Valid options: {', '.join(valid_entities[:10])}\n" + f"Did you mean: {', '.join(suggestions[:3])}?" + ) + + return entity + + +def validate_year( + year: Optional[int], + min_year: int = 1900, + allow_future: bool = False +) -> int: + """ + Validate year parameter. + + Args: + year: Year to validate (None returns current year) + min_year: Minimum valid year + allow_future: Whether future years are allowed + + Returns: + int: Validated year + + Raises: + ValidationError: If year is invalid + + Example: + >>> validate_year(2025) + 2025 + >>> validate_year(None) + 2025 # Current year + """ + current_year = datetime.now().year + + if year is None: + return current_year + + if not isinstance(year, int): + raise ValidationError(f"Year must be integer, got {type(year)}") + + if year < min_year: + raise ValidationError( + f"Year {year} is too old (minimum: {min_year})" + ) + + if not allow_future and year > current_year: + raise ValidationError( + f"Year {year} is in the future (current: {current_year})" + ) + + return year + + +def validate_state(state: str, country: str = "US") -> str: + """Validate state/region parameter.""" + # Country-specific validation + # ... + return state.upper() + + +# Add more validators for domain-specific parameters... +``` + +**Template 2: data_validator.py** + +```python +#!/usr/bin/env python3 +""" +Data validators for {skill-name}. +Validates API responses and analysis outputs. +""" + +import pandas as pd +from typing import Dict, List, Any +from dataclasses import dataclass +from enum import Enum + + +class ValidationLevel(Enum): + """Severity levels for validation results.""" + CRITICAL = "critical" # Must fix + WARNING = "warning" # Should review + INFO = "info" # FYI + + +@dataclass +class ValidationResult: + """Single validation check result.""" + check_name: str + level: ValidationLevel + passed: bool + message: str + details: Optional[Dict] = None + + +class ValidationReport: + """Collection of validation results.""" + + def __init__(self): + self.results: List[ValidationResult] = [] + + def add(self, result: ValidationResult): + """Add validation result.""" + self.results.append(result) + + def has_critical_issues(self) -> bool: + """Check if any critical issues found.""" + return any( + r.level == ValidationLevel.CRITICAL and not r.passed + for r in self.results + ) + + def all_passed(self) -> bool: + """Check if all validations passed.""" + return all(r.passed for r in self.results) + + def get_warnings(self) -> List[str]: + """Get all warning messages.""" + return [ + r.message for r in self.results + if r.level == ValidationLevel.WARNING and not r.passed + ] + + def get_summary(self) -> str: + """Get summary of validation results.""" + total = len(self.results) + passed = sum(1 for r in self.results if r.passed) + critical = sum( + 1 for r in self.results + if r.level == ValidationLevel.CRITICAL and not r.passed + ) + + return ( + f"Validation: {passed}/{total} passed " + f"({critical} critical issues)" + ) + + +class DataValidator: + """Validates API responses and DataFrames.""" + + def validate_response(self, data: Any) -> ValidationReport: + """ + Validate raw API response. + + Args: + data: Raw API response + + Returns: + ValidationReport with results + """ + report = ValidationReport() + + # Check 1: Not empty + report.add(ValidationResult( + check_name="not_empty", + level=ValidationLevel.CRITICAL, + passed=bool(data), + message="Data is empty" if not data else "Data present" + )) + + # Check 2: Correct type + expected_type = (list, dict) + is_correct_type = isinstance(data, expected_type) + report.add(ValidationResult( + check_name="correct_type", + level=ValidationLevel.CRITICAL, + passed=is_correct_type, + message=f"Expected {expected_type}, got {type(data)}" + )) + + # Check 3: Has expected structure + if isinstance(data, dict): + has_data_key = 'data' in data + report.add(ValidationResult( + check_name="has_data_key", + level=ValidationLevel.WARNING, + passed=has_data_key, + message="Response has 'data' key" if has_data_key else "No 'data' key" + )) + + return report + + def validate_dataframe(self, df: pd.DataFrame, data_type: str) -> ValidationReport: + """ + Validate parsed DataFrame. + + Args: + df: Parsed DataFrame + data_type: Type of data (for type-specific checks) + + Returns: + ValidationReport + """ + report = ValidationReport() + + # Check 1: Not empty + report.add(ValidationResult( + check_name="not_empty", + level=ValidationLevel.CRITICAL, + passed=len(df) > 0, + message=f"DataFrame has {len(df)} rows" + )) + + # Check 2: Required columns + required = ['entity', 'year'] # Customize per type + missing = set(required) - set(df.columns) + report.add(ValidationResult( + check_name="required_columns", + level=ValidationLevel.CRITICAL, + passed=len(missing) == 0, + message=f"Missing columns: {missing}" if missing else "All required columns present" + )) + + # Check 3: No excessive NaN values + if len(df) > 0: + nan_pct = (df.isna().sum() / len(df) * 100).max() + report.add(ValidationResult( + check_name="nan_threshold", + level=ValidationLevel.WARNING, + passed=nan_pct < 30, + message=f"Max NaN: {nan_pct:.1f}% ({'OK' if nan_pct < 30 else 'HIGH'})" + )) + + # Check 4: Data types correct + if 'year' in df.columns: + is_int = pd.api.types.is_integer_dtype(df['year']) + report.add(ValidationResult( + check_name="year_type", + level=ValidationLevel.CRITICAL, + passed=is_int, + message="'year' is integer" if is_int else "'year' is not integer" + )) + + return report + + +def validate_{type}_output(result: Dict) -> ValidationReport: + """ + Validate analysis output for {type}. + + Args: + result: Analysis result dict + + Returns: + ValidationReport + """ + report = ValidationReport() + + # Check required keys + required_keys = ['year', 'year_info', 'data'] + for key in required_keys: + report.add(ValidationResult( + check_name=f"has_{key}", + level=ValidationLevel.CRITICAL, + passed=key in result, + message=f"'{key}' present" if key in result else f"Missing '{key}'" + )) + + # Check data quality + if 'data' in result and result['data']: + report.add(ValidationResult( + check_name="data_not_empty", + level=ValidationLevel.CRITICAL, + passed=True, + message="Data is present" + )) + + return report + + +# Main for testing +if __name__ == "__main__": + print("Testing validators...") + + # Test entity validator + print("\n1. Testing validate_entity():") + try: + entity = validate_entity("corn", ["CORN", "SOYBEANS"]) + print(f" āœ“ Valid: {entity}") + except ValidationError as e: + print(f" āœ— Error: {e}") + + # Test year validator + print("\n2. Testing validate_year():") + year = validate_year(2025) + print(f" āœ“ Valid: {year}") + + # Test DataValidator + print("\n3. Testing DataValidator:") + validator = DataValidator() + sample_data = [{'entity': 'CORN', 'year': 2025}] + report = validator.validate_response(sample_data) + print(f" {report.get_summary()}") +``` + +**Template 3: temporal_validator.py** + +```python +#!/usr/bin/env python3 +""" +Temporal validators for {skill-name}. +Checks temporal consistency and data age. +""" + +import pandas as pd +from datetime import datetime, timedelta +from typing import List +from .data_validator import ValidationResult, ValidationReport, ValidationLevel + + +def validate_temporal_consistency(df: pd.DataFrame) -> ValidationReport: + """ + Check temporal consistency in data. + + Validations: + - No future dates + - Years in valid range + - No suspicious gaps in time series + - Data age is acceptable + + Args: + df: DataFrame with 'year' column + + Returns: + ValidationReport + """ + report = ValidationReport() + current_year = datetime.now().year + + if 'year' not in df.columns: + report.add(ValidationResult( + check_name="has_year_column", + level=ValidationLevel.CRITICAL, + passed=False, + message="Missing 'year' column" + )) + return report + + # Check 1: No future years + max_year = df['year'].max() + report.add(ValidationResult( + check_name="no_future_years", + level=ValidationLevel.CRITICAL, + passed=max_year <= current_year, + message=f"Max year: {max_year} ({'valid' if max_year <= current_year else 'FUTURE!'})" + )) + + # Check 2: Years in reasonable range + min_year = df['year'].min() + is_reasonable = min_year >= 1900 + report.add(ValidationResult( + check_name="reasonable_year_range", + level=ValidationLevel.WARNING, + passed=is_reasonable, + message=f"Year range: {min_year}-{max_year}" + )) + + # Check 3: Data age (is data recent enough?) + data_age_years = current_year - max_year + is_recent = data_age_years <= 2 + report.add(ValidationResult( + check_name="data_freshness", + level=ValidationLevel.WARNING, + passed=is_recent, + message=f"Data age: {data_age_years} years ({'recent' if is_recent else 'STALE'})" + )) + + # Check 4: No suspicious gaps in time series + if len(df['year'].unique()) > 2: + years_sorted = sorted(df['year'].unique()) + gaps = [ + years_sorted[i+1] - years_sorted[i] + for i in range(len(years_sorted)-1) + ] + max_gap = max(gaps) if gaps else 0 + has_large_gap = max_gap > 2 + + report.add(ValidationResult( + check_name="no_large_gaps", + level=ValidationLevel.WARNING, + passed=not has_large_gap, + message=f"Max gap: {max_gap} years" + (" (suspicious)" if has_large_gap else "") + )) + + return report + + +def validate_week_number(week: int, year: int) -> ValidationResult: + """Validate week number is in valid range for year.""" + # Most data types use weeks 1-53 + is_valid = 1 <= week <= 53 + + return ValidationResult( + check_name="valid_week", + level=ValidationLevel.CRITICAL, + passed=is_valid, + message=f"Week {week} ({'valid' if is_valid else 'INVALID: must be 1-53'})" + ) + + +# Add more temporal validators as needed... +``` + +**Template 4: completeness_validator.py** + +```python +#!/usr/bin/env python3 +""" +Completeness validators for {skill-name}. +Checks data completeness and coverage. +""" + +import pandas as pd +from typing import List, Set +from .data_validator import ValidationResult, ValidationReport, ValidationLevel + + +def validate_completeness( + df: pd.DataFrame, + expected_entities: Optional[List[str]] = None, + expected_years: Optional[List[int]] = None +) -> ValidationReport: + """ + Validate data completeness. + + Args: + df: DataFrame to validate + expected_entities: Expected entities (None to skip) + expected_years: Expected years (None to skip) + + Returns: + ValidationReport + """ + report = ValidationReport() + + # Check 1: All expected entities present + if expected_entities: + actual_entities = set(df['entity'].unique()) + expected_set = set(expected_entities) + missing = expected_set - actual_entities + + report.add(ValidationResult( + check_name="all_entities_present", + level=ValidationLevel.WARNING, + passed=len(missing) == 0, + message=f"Missing entities: {missing}" if missing else "All entities present", + details={'missing': list(missing)} + )) + + # Check 2: All expected years present + if expected_years: + actual_years = set(df['year'].unique()) + expected_set = set(expected_years) + missing = expected_set - actual_years + + report.add(ValidationResult( + check_name="all_years_present", + level=ValidationLevel.WARNING, + passed=len(missing) == 0, + message=f"Missing years: {missing}" if missing else "All years present" + )) + + # Check 3: No excessive nulls in critical columns + critical_columns = ['entity', 'year'] # Customize + for col in critical_columns: + if col in df.columns: + null_count = df[col].isna().sum() + report.add(ValidationResult( + check_name=f"{col}_no_nulls", + level=ValidationLevel.CRITICAL, + passed=null_count == 0, + message=f"'{col}' has {null_count} nulls" + )) + + # Check 4: Coverage percentage + if expected_entities and expected_years: + expected_total = len(expected_entities) * len(expected_years) + actual_total = len(df) + coverage_pct = (actual_total / expected_total) * 100 if expected_total > 0 else 0 + + report.add(ValidationResult( + check_name="coverage_percentage", + level=ValidationLevel.INFO, + passed=coverage_pct >= 80, + message=f"Coverage: {coverage_pct:.1f}% ({actual_total}/{expected_total})" + )) + + return report +``` + +**Integration in analysis functions:** + +```python +def {analysis_function}(entity: str, year: Optional[int] = None, ...) -> Dict: + """Analysis function with validation.""" + from utils.validators.parameter_validator import validate_entity, validate_year + from utils.validators.data_validator import DataValidator + from utils.validators.temporal_validator import validate_temporal_consistency + + # VALIDATE INPUTS (before doing anything!) + entity = validate_entity(entity, valid_entities=[...]) + year = validate_year(year) + + # Fetch data + data = fetch_{metric}(entity, year) + + # VALIDATE API RESPONSE + validator = DataValidator() + response_validation = validator.validate_response(data) + + if response_validation.has_critical_issues(): + raise DataQualityError( + f"API response validation failed: {response_validation.get_summary()}" + ) + + # Parse + df = parse_{type}(data) + + # VALIDATE PARSED DATA + df_validation = validator.validate_dataframe(df, '{type}') + temporal_validation = validate_temporal_consistency(df) + + if df_validation.has_critical_issues(): + raise DataQualityError( + f"Data validation failed: {df_validation.get_summary()}" + ) + + # Analyze + results = analyze(df) + + # Return with validation info + return { + 'data': results, + 'year': year, + 'year_info': format_year_message(year, year_requested), + 'validation': { + 'passed': df_validation.all_passed(), + 'warnings': df_validation.get_warnings(), + 'report': df_validation.get_summary() + } + } +``` + +**Impact:** +- āœ… Reliable data (validated at multiple layers) +- āœ… Transparency (user sees validation report) +- āœ… Clear error messages (not just "generic error") +- āœ… Problem detection (gaps, nulls, inconsistencies) + +--- + +**4. Write references** + +For each reference file: + +- 1000-2000 words +- Useful content (examples, methodologies, guides) +- Well structured (headings, lists, code blocks) +- Well-formatted markdown + +**5. Create assets** + +- Syntactically valid JSONs +- Real values with comments +- Logical structure + +**6. Write README.md** + +- Step-by-step installation +- Required configuration +- Usage examples +- Troubleshooting + +**7. Create DECISIONS.md** + +Document all decisions made: + +- Which API chosen and why +- Which analyses defined and justification +- Structure chosen and rationale +- Trade-offs considered + +**8. Create VERSION and CHANGELOG.md** (šŸ†• Enhancement #7 - Versioning) + +**8.1 Create VERSION file:** + +``` +1.0.0 +``` + +**8.2 Create CHANGELOG.md:** + +```markdown +# Changelog + +All notable changes to {skill-name} will be documented here. + +Format based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). +Versioning follows [Semantic Versioning](https://semver.org/). + +## [1.0.0] - {current_date} + +### Added + +**Core Functionality:** +- {function1}: {Description of what it does} +- {function2}: {Description of what it does} +- {function3}: {Description of what it does} +... + +**Data Sources:** +- {API_name}: {Coverage description} +- Authentication: {auth_method} +- Rate limit: {limit} + +**Analysis Capabilities:** +- {analysis1}: {Description and methodology} +- {analysis2}: {Description and methodology} +... + +**Utilities:** +- Cache system with {TTL} TTL +- Rate limiting: {limit} per {period} +- Error handling with automatic retries +- Data validation and quality checks + +### Data Coverage + +**Metrics implemented:** +- {metric1}: {Coverage details} +- {metric2}: {Coverage details} +... + +**Geographic coverage:** {geo_coverage} +**Temporal coverage:** {temporal_coverage} + +### Known Limitations + +- {limitation1} +- {limitation2} +... + +### Planned for v2.0 + +- {planned_feature1} +- {planned_feature2} +... + +## [Unreleased] + +### Planned + +- Add support for {feature} +- Improve performance for {scenario} +- Expand coverage to {new_area} +``` + +**8.3 Update marketplace.json with version:** + +Edit `.claude-plugin/marketplace.json` to include: + +```json +{ + "metadata": { + "description": "...", + "version": "1.0.0", + "created": "{current_date}", + "updated": "{current_date}" + } +} +``` + +**8.4 Create .bumpversion.cfg (optional):** + +If you want version automation: + +```ini +[bumpversion] +current_version = 1.0.0 +commit = False +tag = False + +[bumpversion:file:VERSION] + +[bumpversion:file:.claude-plugin/marketplace.json] +search = "version": "{current_version}" +replace = "version": "{new_version}" + +[bumpversion:file:CHANGELOG.md] +search = ## [Unreleased] +replace = ## [Unreleased] + +## [{new_version}] - {now:%Y-%m-%d} +``` + +**Impact:** +- āœ… Change traceability +- āœ… Professionalism +- āœ… Facilitates future updates +- āœ… Users know what changed between versions + +**9. Create INSTALLATION.md** (Didactic Tutorial) + +[Content of INSTALLATION.md as previously specified] + +### Practical Implementation + +**Create agent in subdirectory**: + +```bash +# Agent name based on domain/objective +agent_name="nass-usda-agriculture" # example + +# Create structure +mkdir -p $agent_name/{scripts/utils,references,assets,data} + +# Implement each file +# [Claude creates each file with Write tool] +``` + +**At the end, inform user**: + +``` +āœ… Agent created in ./{agent_name}/ + +šŸ“ Structure: +- .claude-plugin/marketplace.json āœ… (installation + version) +- SKILL.md (6,200 words) +- scripts/ (2,500+ lines of code) + ā”œā”€ utils/helpers.py āœ… (temporal context) + ā”œā”€ utils/validators/ āœ… (4 validators, ~800 lines) + ā”œā”€ parse_{type}*.py āœ… (1 per data type, modular) + ā””ā”€ comprehensive_{domain}_report() āœ… +- tests/ (25+ tests, ~800 lines) āœ… + ā”œā”€ test_integration.py (end-to-end) + ā”œā”€ test_parse.py (all parsers) + ā”œā”€ test_helpers.py (temporal) + ā””ā”€ test_validation.py (validators) +- references/ (5,000 words) +- assets/ (2 configs) +- README.md (1,000+ words with Testing section) +- INSTALLATION.md (1,500 words) āœ… +- DECISIONS.md (justifications) +- VERSION (1.0.0) āœ… +- CHANGELOG.md (release notes) āœ… + +šŸš€ To install: +/plugin marketplace add ./{agent_name} + +šŸ’” Usage examples: +"[example 1]" +"[example 2]" +``` + +**See** `references/phase5-implementation.md` for complete implementation guide. + +## Complete Flow: Step-by-Step + +### User Input + +User describes workflow/objective: + +``` +"Every day I download US crop data from USDA, +compare current year vs previous, create state ranking +by production, and generate report. Takes 2 hours. +Automate this." +``` + +### Autonomous Execution + +**Claude executes internally** (doesn't need to show everything): + +**PHASE 1**: Discovery (~5-10 min) + +``` +[WebSearch: "USDA API free agriculture data"] +[WebFetch: NASS API docs] +[Compare: NASS vs ERS vs FAO] +ā†’ DECISION: NASS QuickStats API + Justification: Free, data since 1866, all crops, state-level +``` + +**PHASE 2**: Design (~5 min) + +``` +[Analyze: "compare years", "state ranking", "generate report"] +ā†’ DEFINE analyses: + 1. YoY comparison (compare years) + 2. State ranking (state ranking) + 3. Trend analysis (if user asks for trends) + 4. Report generator (generate report) +``` + +**PHASE 3**: Architecture (~5 min) + +``` +ā†’ STRUCTURE: + scripts/ + - fetch_nass.py (API requests) + - parse_nass.py (parsing) + - analyze_nass.py (YoY, ranking, trends) + - generate_report.py (report) + utils/ + - cache_manager.py + - rate_limiter.py + references/ + - nass-api-guide.md + - analysis-methods.md + - report-templates.md + assets/ + - config.json + - metadata.json +``` + +**PHASE 4**: Detection (~2 min) + +``` +ā†’ KEYWORDS: + Entities: USDA, NASS, agriculture, crops + Commodities: corn, soybeans, wheat + Metrics: production, area, yield + Actions: compare, ranking, trend + Geography: US, states, Iowa, Illinois + +ā†’ DESCRIPTION: "This skill should be used when user + asks about US agriculture, production, area, + crop yield..." +``` + +**PHASE 5**: Implementation (~45 min) + +``` +[Write: SKILL.md] āœ… 6,000 words +[Write: fetch_nass.py] āœ… 280 lines +[Write: parse_nass.py] āœ… 180 lines +[Write: analyze_nass.py] āœ… 520 lines +[Write: generate_report.py] āœ… 200 lines +[Write: utils/cache_manager.py] āœ… 120 lines +[Write: utils/rate_limiter.py] āœ… 110 lines +[Write: references/nass-api-guide.md] āœ… 1,500 words +[Write: references/analysis-methods.md] āœ… 2,000 words +[Write: references/report-templates.md] āœ… 1,000 words +[Write: assets/config.json] āœ… +[Write: assets/metadata.json] āœ… +[Write: README.md] āœ… 800 words +[Write: DECISIONS.md] āœ… 600 words +``` + +### User Output + +``` +āœ… Agent created: ./usda-agriculture-agent/ + +šŸ“Š Technical Decisions: +- API: USDA NASS QuickStats (free, data since 1866) +- Analyses: YoY, Ranking, Trends, Report Generation +- Structure: 4 main scripts + 2 utils +- Cache: Historical permanent, current 7 days + +šŸ“ Files Created: +- SKILL.md: 6,000 words +- Python code: 2,500+ lines + ā”œā”€ Modular parsers (1 per type) + ā”œā”€ Validation system (800 lines) + ā”œā”€ Temporal helpers + ā””ā”€ comprehensive_report() +- Tests: 25+ tests (800 lines) +- References: 4,500 words +- Configs: 2 files +- README: Complete (with Testing) +- INSTALLATION.md: Didactic tutorial +- VERSION: 1.0.0 +- CHANGELOG.md: Complete release notes + +šŸŽÆ Estimated Savings: +- Before: 2 hours/day +- After: 3 minutes/day +- Savings: 98.5% (117h/month ā†’ 1.5h/month) + +šŸš€ To install and use: + +# 1. Get API key (free): +Visit: https://quickstats.nass.usda.gov/api#registration + +# 2. Configure: +export NASS_API_KEY="your_key_here" + +# 3. Install skill: +/plugin marketplace add ./usda-agriculture-agent + +# 4. Use (examples): +"US corn production in 2023" +"Compare soybeans this year vs last year" +"Ranking of wheat producing states" +"Generate current crop report" +``` + +## Detailed References + +For details of each phase, load references: + +- `references/phase1-discovery.md`: API research and decision +- `references/phase2-design.md`: Analysis definition +- `references/phase3-architecture.md`: Project structuring +- `references/phase4-detection.md`: Keywords and automatic activation +- `references/phase5-implementation.md`: Code implementation +- `references/quality-standards.md`: Mandatory standards +- `references/examples.md`: Complete examples of created agents + +## Meta-Skill Usage Examples + +### Example 1: Simple Workflow + +``` +šŸ‘¤ "Automate: download weather data, calculate averages, + generate chart. I do this every week, takes 1h." + +šŸ¤– [Activates agent-creator] + [Phase 1]: Research ā†’ NOAA API + [Phase 2]: Defines ā†’ Download, Aggregation, Visualization + [Phase 3]: Structure ā†’ 3 scripts + 2 utils + [Phase 4]: Keywords ā†’ weather, climate, temperature, NOAA + [Phase 5]: Implements everything + + āœ… Agent created: ./weather-analysis-agent/ + [Installation instructions] +``` + +### Example 2: Complex Workflow + +``` +šŸ‘¤ "I need an agent for stock financial analysis: + fetch prices, calculate technical indicators (RSI, MACD), + compare with benchmarks, generate alerts. Daily, 3h." + +šŸ¤– [Activates agent-creator] + [Phase 1]: Research ā†’ Alpha Vantage API (or Yahoo Finance) + [Phase 2]: Defines ā†’ Price fetching, Technical indicators, + Benchmark comparison, Alert system + [Phase 3]: Structure ā†’ 5 scripts + 4 utils + [Phase 4]: Keywords ā†’ stock, RSI, MACD, technical analysis + [Phase 5]: Implements + + āœ… Agent created: ./stock-technical-analysis-agent/ +``` + +### Example 3: Just Vague Description + +``` +šŸ‘¤ "Create an agent for Brazilian agriculture" + +šŸ¤– [Activates agent-creator] + + "Got it! I'll create an agent for Brazilian agriculture. + + A few quick questions: + - Focus on which crops? (soy, corn, coffee, all?) + - Desired analyses? (production, prices, exports?) + - Preferred source? (CONAB, IBGE, both?) + + Or I can decide based on most common use cases + (grain production via CONAB). Prefer I decide?" + +šŸ‘¤ "You decide, focus on grain production." + +šŸ¤– [Phase 1]: CONAB Grain Survey + [Phase 2]: YoY, Ranking, Regional, Trends + [Phase 3-5]: Creates everything + + āœ… Agent created: ./conab-agriculture-agent/ +``` + +--- + +## PHASE 6: Test Suite Generation (šŸ†• Enhancement #4 - MANDATORY!) + +**Objective**: Generate comprehensive test suite that validates ALL functions + +**āš ļø COMMON PROBLEM:** v1.0 without tests. Difficult to validate code works, impossible to do regression testing. + +**Solution:** Automatically generate 25+ tests covering all layers! + +### Test Structure + +``` +tests/ +ā”œā”€ā”€ __init__.py +ā”œā”€ā”€ test_fetch.py # Test API fetch functions +ā”œā”€ā”€ test_parse.py # Test each parser +ā”œā”€ā”€ test_analyze.py # Test analysis functions +ā”œā”€ā”€ test_integration.py # End-to-end tests +ā”œā”€ā”€ test_validation.py # Test validators +ā”œā”€ā”€ test_helpers.py # Test temporal helpers +ā””ā”€ā”€ conftest.py # Shared fixtures (pytest) +``` + +### Template 1: test_integration.py (MAIN!) + +```python +#!/usr/bin/env python3 +""" +Integration tests for {skill-name}. +Tests complete workflows from query to result. +""" + +import sys +from pathlib import Path + +# Add scripts to path +sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts')) + +from analyze_{domain} import ( + {function1}, + {function2}, + {function3}, + comprehensive_{domain}_report +) + + +def test_{function1}_basic(): + """Test {function1} with auto-year detection.""" + print(f"\nāœ“ Testing {function1}()...") + + try: + # Test auto-year detection (year=None) + result = {function1}('{example_entity}') + + # Validations + assert 'year' in result, "Missing 'year' in result" + assert 'year_info' in result, "Missing 'year_info'" + assert 'data' in result, "Missing 'data'" + assert result['year'] >= 2024, f"Year too old: {result['year']}" + + print(f" āœ“ Auto-year working: {result['year']}") + print(f" āœ“ Year info: {result['year_info']}") + print(f" āœ“ Data present: {len(result.get('data', {}))} fields") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + import traceback + traceback.print_exc() + return False + + +def test_{function1}_specific_year(): + """Test {function1} with specific year.""" + print(f"\nāœ“ Testing {function1}(year=2024)...") + + try: + result = {function1}('{example_entity}', year=2024) + + assert result['year'] == 2024, "Requested year not used" + assert result['year_requested'] == 2024, "year_requested not tracked" + + print(f" āœ“ Specific year working: {result['year']}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + return False + + +def test_{function2}_comparison(): + """Test {function2} (comparison function).""" + print(f"\nāœ“ Testing {function2}()...") + + try: + result = {function2}('{example_entity}', year1=2024, year2=2023) + + # Validations specific to comparison + assert 'change_percent' in result, "Missing 'change_percent'" + assert isinstance(result['change_percent'], (int, float)), "change_percent not numeric" + + print(f" āœ“ Comparison working: {result.get('change_percent')}% change") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + return False + + +def test_comprehensive_report(): + """Test comprehensive report (all-in-one function).""" + print(f"\nāœ“ Testing comprehensive_{domain}_report()...") + + try: + result = comprehensive_{domain}_report('{example_entity}') + + # Validations + assert 'metrics' in result, "Missing 'metrics'" + assert 'summary' in result, "Missing 'summary'" + assert 'alerts' in result, "Missing 'alerts'" + assert isinstance(result['metrics'], dict), "'metrics' must be dict" + + metrics_count = len(result['metrics']) + print(f" āœ“ Comprehensive report working") + print(f" āœ“ Metrics combined: {metrics_count}") + print(f" āœ“ Summary: {result['summary'][:100]}...") + print(f" āœ“ Alerts: {len(result['alerts'])}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + return False + + +def test_validation_integration(): + """Test that validation is integrated in functions.""" + print(f"\nāœ“ Testing validation integration...") + + try: + result = {function1}('{example_entity}') + + # Check validation info is present + assert 'validation' in result, "Missing 'validation' info" + assert 'passed' in result['validation'], "Missing validation.passed" + assert 'report' in result['validation'], "Missing validation.report" + + print(f" āœ“ Validation present: {result['validation']['report']}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + return False + + +def main(): + """Run all integration tests.""" + print("=" * 70) + print("INTEGRATION TESTS - {skill-name}") + print("=" * 70) + + tests = [ + ("Auto-year detection", test_{function1}_basic), + ("Specific year", test_{function1}_specific_year), + ("Comparison function", test_{function2}_comparison), + ("Comprehensive report", test_comprehensive_report), + ("Validation integration", test_validation_integration), + ] + + results = [] + for test_name, test_func in tests: + passed = test_func() + results.append((test_name, passed)) + + # Summary + print("\n" + "=" * 70) + print("SUMMARY") + print("=" * 70) + + for test_name, passed in results: + status = "āœ… PASS" if passed else "āŒ FAIL" + print(f"{status}: {test_name}") + + passed_count = sum(1 for _, p in results if p) + total_count = len(results) + + print(f"\nResults: {passed_count}/{total_count} passed") + + return passed_count == total_count + + +if __name__ == "__main__": + success = main() + sys.exit(0 if success else 1) +``` + +### Template 2: test_parse.py + +```python +#!/usr/bin/env python3 +"""Tests for parsers.""" + +import sys +from pathlib import Path +import pandas as pd + +sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts')) + +from parse_{type1} import parse_{type1}_response +from parse_{type2} import parse_{type2}_response +# Import all parsers... + + +def test_parse_{type1}(): + """Test {type1} parser.""" + print("\nāœ“ Testing parse_{type1}_response()...") + + sample_data = [ + {'{field1}': 'VALUE1', '{field2}': 2025, '{field3}': '1,234,567'} + ] + + try: + df = parse_{type1}_response(sample_data) + + # Validations + assert isinstance(df, pd.DataFrame), "Must return DataFrame" + assert len(df) == 1, f"Expected 1 row, got {len(df)}" + assert 'entity' in df.columns, "Missing 'entity' column" + assert 'year' in df.columns, "Missing 'year' column" + + print(f" āœ“ Parsed: {len(df)} records") + print(f" āœ“ Columns: {list(df.columns)}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + return False + + +# Repeat for all parsers... + +def main(): + """Run parser tests.""" + tests = [ + test_parse_{type1}, + test_parse_{type2}, + # Add all... + ] + + passed = sum(1 for test in tests if test()) + print(f"\nResults: {passed}/{len(tests)} passed") + + return passed == len(tests) + + +if __name__ == "__main__": + sys.exit(0 if main() else 1) +``` + +### Template 3: test_helpers.py + +```python +#!/usr/bin/env python3 +"""Tests for temporal helpers.""" + +import sys +from pathlib import Path +from datetime import datetime + +sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts')) + +from utils.helpers import ( + get_current_{domain}_year, + get_{domain}_year_with_fallback, + should_try_previous_year, + format_year_message +) + + +def test_get_current_year(): + """Test current year detection.""" + year = get_current_{domain}_year() + current = datetime.now().year + + assert year == current, f"Expected {current}, got {year}" + print(f"āœ“ Current year: {year}") + return True + + +def test_year_with_fallback(): + """Test year fallback logic.""" + primary, fallback = get_{domain}_year_with_fallback(2024) + + assert primary == 2024, "Primary should be 2024" + assert fallback == 2023, "Fallback should be 2023" + + print(f"āœ“ Fallback: {primary} ā†’ {fallback}") + return True + + +def test_format_year_message(): + """Test year message formatting.""" + msg = format_year_message(2024, 2025) + + assert '2024' in msg, "Must mention year used" + assert '2025' in msg, "Must mention year requested" + + print(f"āœ“ Message: {msg}") + return True + + +def main(): + """Run helper tests.""" + tests = [ + test_get_current_year, + test_year_with_fallback, + test_format_year_message + ] + + passed = sum(1 for test in tests if test()) + print(f"\nResults: {passed}/{len(tests)} passed") + + return passed == len(tests) + + +if __name__ == "__main__": + sys.exit(0 if main() else 1) +``` + +### Minimum Test Coverage + +**For skill to be considered complete, needs:** + +- [ ] test_integration.py with ā‰„5 end-to-end tests +- [ ] test_parse.py with 1 test per parser +- [ ] test_analyze.py with 1 test per analysis function +- [ ] test_helpers.py with ā‰„3 tests +- [ ] test_validation.py with ā‰„5 tests +- [ ] **Total:** ā‰„25 tests +- [ ] **Coverage:** ā‰„80% of code +- [ ] **All tests PASS** + +### How to test + +Include in README.md: + +```markdown +## Testing + +### Run All Tests + +```bash +cd {skill-name} +python3 tests/test_integration.py +``` + +### Run Specific Tests + +```bash +python3 tests/test_parse.py +python3 tests/test_helpers.py +python3 tests/test_validation.py +``` + +### Expected Output + +``` +====================================================================== +INTEGRATION TESTS - {skill-name} +====================================================================== + +āœ“ Testing {function1}()... + āœ“ Auto-year working: 2025 + āœ“ Data present: 8 fields + +āœ“ Testing {function2}()... + āœ“ Comparison working: +12.3% change + +... + +====================================================================== +SUMMARY +====================================================================== +āœ… PASS: Auto-year detection +āœ… PASS: Specific year +āœ… PASS: Comparison function +āœ… PASS: Comprehensive report +āœ… PASS: Validation integration + +Results: 5/5 passed +``` +``` + +### Test Suite Benefits + +- āœ… Reliability: Tested and working code +- āœ… Regression testing: Detects breaks when modifying +- āœ… Executable documentation: Tests show how to use +- āœ… CI/CD ready: Can run automatically +- āœ… Professionalism: Production-quality skills + +**Impact:** Generated skills are tested and reliable from v1.0! + +--- + +## Agent Creation Workflow: Checklist + +When creating an agent, follow this checklist RIGOROUSLY in order: + +--- + +### šŸšØ STEP 0: MANDATORY - FIRST STEP + +**Execute BEFORE anything else:** + +- [ ] šŸšØ Create `.claude-plugin/marketplace.json` +- [ ] šŸšØ Validate JSON syntax with python +- [ ] šŸšØ Verify mandatory fields filled +- [ ] šŸšØ Confirm: "Marketplace.json created and validated - can proceed" + +**šŸ›‘ DO NOT PROCEED without completing ALL items above!** + +--- + +### āœ… Phase 1-4: Planning + +- [ ] Domain identified +- [ ] API researched and decided (with justification) +- [ ] **API completeness analysis** (Phase 1.6 - coverage ā‰„50%) +- [ ] Analyses defined (4-6 main + comprehensive_report) +- [ ] Structure planned (modular parsers, validators/) +- [ ] Keywords determined (ā‰„60 unique) + +--- + +### āœ… Phase 5: Implementation + +- [ ] .claude-plugin/marketplace.json created FIRST +- [ ] marketplace.json validated (syntax + fields) +- [ ] SKILL.md created with correct frontmatter +- [ ] **CRITICAL:** SKILL.md description copied to marketplace.json ā†’ plugins[0].description (IDENTICAL!) +- [ ] Validate synchronization: SKILL.md description === marketplace.json +- [ ] **MANDATORY:** utils/helpers.py created (temporal context) +- [ ] **MANDATORY:** utils/validators/ created (4 validators) +- [ ] **MANDATORY:** Modular parsers (1 per data type) +- [ ] **MANDATORY:** comprehensive_{domain}_report() implemented +- [ ] DECISIONS.md documenting choices +- [ ] VERSION file created (e.g., 1.0.0) +- [ ] CHANGELOG.md created with complete v1.0.0 entry +- [ ] marketplace.json with version field +- [ ] Implement functional code (no TODOs) +- [ ] Write complete docstrings +- [ ] Add error handling +- [ ] Write references with useful content +- [ ] Create real configs +- [ ] Write complete README +- [ ] INSTALLATION.md with complete tutorial + +--- + +### āœ… Phase 6: Test Suite + +- [ ] tests/ directory created +- [ ] test_integration.py with ā‰„5 end-to-end tests +- [ ] test_parse.py with 1 test per parser +- [ ] test_analyze.py with 1 test per analysis function +- [ ] test_helpers.py with ā‰„3 tests +- [ ] test_validation.py with ā‰„5 tests +- [ ] **Total:** ā‰„25 tests implemented +- [ ] **ALL tests PASS** (execute and validate!) +- [ ] "Testing" section added to README.md + +--- + +### āœ… Final Validation + +- [ ] Validate marketplace.json again (syntax + synchronized description) +- [ ] Validate other JSONs (configs, assets) +- [ ] Verify imports work +- [ ] Check no placeholder/TODO +- [ ] Test main logic manually +- [ ] Verify README has all instructions +- [ ] Calculate estimated ROI (time before vs after) + +--- + +### šŸš€ MANDATORY TEST - DO NOT SKIP THIS STEP! + +**Execute this command MANDATORY before delivering:** + +```bash +cd /path/to/skills +/plugin marketplace add ./agent-name +``` + +**Verifications:** + +- [ ] āœ… Command executed without errors +- [ ] āœ… Skill appears in installed plugins list +- [ ] āœ… Claude recognizes the skill (do test question) + +**šŸ›‘ If test fails:** + +1. Verify marketplace.json exists +2. Verify JSON is valid +3. Verify description is synchronized +4. Fix and test again + +**Only deliver to user AFTER installation test passes!** + +--- + +### āœ… Deliver to User + +- [ ] Show created structure +- [ ] Summarize main decisions +- [ ] List files and sizes +- [ ] Give installation instructions (command tested above) +- [ ] Give 3-5 usage examples +- [ ] Inform estimated ROI +- [ ] **Confirm: "Skill tested and installed successfully"** + +## User Communication + +### During Creation + +**Show high-level progress**: + +``` +šŸ” Phase 1: Researching APIs... + āœ“ 5 options found + āœ“ Decided: NASS API (free, complete data) + +šŸŽØ Phase 2: Defining analyses... + āœ“ 15 typical questions identified + āœ“ 5 main analyses defined + +šŸ—ļø Phase 3: Structuring project... + āœ“ 3 scripts + 2 utils planned + +šŸŽÆ Phase 4: Defining detection... + āœ“ 50+ keywords identified + +āš™ļø Phase 5: Implementing code... + [Progress while creating files] + āœ“ SKILL.md (6,200 words) + āœ“ fetch_nass.py (280 lines) + āœ“ parse_nass.py (180 lines) + [...] +``` + +**Don't show**: Technical details during creation (code blocks, etc). Just progress. + +### After Completion + +**Executive summary**: + +``` +āœ… AGENT CREATED SUCCESSFULLY! + +šŸ“‚ Location: ./usda-agriculture-agent/ + +šŸ“Š Main Decisions: +- API: USDA NASS QuickStats +- Analyses: YoY, Ranking, Trends, Reports +- Implementation: 1,410 lines Python + 4,500 words docs + +šŸ’° Estimated ROI: +- Time before: 2h/day +- Time after: 3min/day +- Savings: 117h/month + +šŸŽ“ See DECISIONS.md for complete justifications. + +šŸš€ NEXT STEPS: + +1. Get API key (free): + https://quickstats.nass.usda.gov/api#registration + +2. Configure: + export NASS_API_KEY="your_key" + +3. Install: + /plugin marketplace add ./usda-agriculture-agent + +4. Test: + "US corn production in 2023" + "Compare soybeans this year vs last year" + +See README.md for complete instructions. +``` + +## Keywords for This Meta-Skill Detection + +This meta-skill (agent-creator) is activated when user mentions: + +**Create/Develop**: + +- "Create an agent" +- "Develop agent" +- "Create skill" +- "Develop skill" +- "Build agent" + +**Automate**: + +- "Automate this workflow" +- "Automate this process" +- "Automate this task" +- "Need to automate" +- "Turn into agent" + +**Repetitive Workflow**: + +- "Every day I do" +- "Repeatedly need to" +- "Manual process" +- "Workflow that takes Xh" +- "Task I repeat" + +**Agent for Domain**: + +- "Agent for [domain]" +- "Custom skill for [domain]" +- "Specialize Claude in [domain]" + +## āš ļø Troubleshooting: Common Marketplace.json Errors + +### Error: "Failed to install plugin" + +**Most common cause:** marketplace.json doesn't exist or is poorly formatted + +**Diagnosis:** + +```bash +# 1. Verify file exists +ls -la agent-name/.claude-plugin/marketplace.json + +# 2. Validate JSON +python3 -c "import json; json.load(open('agent-name/.claude-plugin/marketplace.json'))" + +# 3. View content +cat agent-name/.claude-plugin/marketplace.json +``` + +**Solutions:** + +1. If file doesn't exist: Go back to STEP 0 and create +2. If invalid JSON: Fix syntax errors +3. If missing fields: Compare with STEP 0 template + +### Error: "Skill not activating" + +**Cause:** marketplace.json description ā‰  SKILL.md description + +**Diagnosis:** + +```bash +# Compare descriptions +grep "description:" agent-name/SKILL.md +grep "\"description\":" agent-name/.claude-plugin/marketplace.json +``` + +**Solution:** + +1. Copy EXACT description from SKILL.md frontmatter +2. Paste in marketplace.json ā†’ plugins[0].description +3. Ensure they are IDENTICAL (word for word) +4. Save and test installation again + +### Error: "Invalid plugin structure" + +**Cause:** Mandatory marketplace.json fields incorrect + +**Verify:** + +- āœ… `plugins[0].skills` = `["./"]` (not `["SKILL.md"]` or other value) +- āœ… `plugins[0].source` = `"./"` (not empty or other value) +- āœ… `name` in JSON root matches directory name + +**Solution:** +Edit marketplace.json and fix fields above according to STEP 0 template. + +## Limitations and Warnings + +### When NOT to use + +āŒ Don't use this skill for: + +- Editing existing skills (use directly) +- Debugging skills (use directly) +- Questions about skills (answer directly) + +### Warnings + +āš ļø **Creation time**: + +- Simple agents: ~30-60 min +- Complex agents: ~60-120 min +- It's normal to take time (creating everything from scratch) + +āš ļø **Review needed**: + +- Created agent is functional but may need adjustments +- Test examples in README +- Iterate if necessary + +āš ļø **API keys**: + +- User needs to obtain API key +- Instructions in created agent's README diff --git a/references/phase1-discovery.md b/references/phase1-discovery.md new file mode 100644 index 0000000..522f9db --- /dev/null +++ b/references/phase1-discovery.md @@ -0,0 +1,454 @@ +# Phase 1: Discovery and API Research + +## Objective + +Research and **DECIDE** autonomously which API or data source to use for the agent. + +## Detailed Process + +### Step 1: Identify Domain + +From user input, extract the main domain: + +| User Input | Identified Domain | +|------------------|---------------------| +| "US crop data" | Agriculture (US) | +| "stock market analysis" | Finance / Stock Market | +| "global climate data" | Climate / Meteorology | +| "economic indicators" | Economy / Macro | +| "commodity data" | Trading / Commodities | + +### Step 2: Search Available APIs + +For the identified domain, use WebSearch to find public APIs: + +**Search queries**: +``` +"[domain] API free public data" +"[domain] government API documentation" +"best API for [domain] historical data" +"[domain] open data sources" +``` + +**Example (US agriculture)**: +```bash +WebSearch: "US agriculture API free historical data" +WebSearch: "USDA API documentation" +WebSearch: "agricultural statistics API United States" +``` + +**Typical result**: 5-10 candidate APIs + +### Step 3: Research Documentation + +For each candidate API, use WebFetch to load: +- Homepage/overview +- Getting started guide +- API reference +- Rate limits and pricing + +**Extract information**: + +```markdown +## API 1: [Name] + +**URL**: [base URL] +**Docs**: [docs URL] + +**Authentication**: +- Type: API key / OAuth / None +- Cost: Free / Paid +- How to obtain: [steps] + +**Available Data**: +- Temporal coverage: [from when to when] +- Geographic coverage: [countries, regions] +- Metrics: [list] +- Granularity: [daily, monthly, annual] + +**Limitations**: +- Rate limit: [requests per day/hour] +- Max records: [per request] +- Throttling: [yes/no] + +**Quality**: +- Source: [official government / private] +- Reliability: [high/medium/low] +- Update frequency: [frequency] + +**Documentation**: +- Quality: [excellent/good/poor] + +### Step 4: API Capability Inventory (NEW v2.0 - CRITICAL!) + +**OBJECTIVE:** Ensure the skill uses 100% of API capabilities, not just the basics! + +**LEARNING:** us-crop-monitor v1.0 used only CONDITION (1 of 5 NASS metrics). +v2.0 had to add PROGRESS, YIELD, PRODUCTION, AREA (+3,500 lines of rework). + +**Process:** + +**Step 4.1: Complete Inventory** + +For the chosen API, catalog ALL data types: + +```markdown +## Complete Inventory - {API Name} + +**Available Metrics/Endpoints:** + +| Endpoint/Metric | Returns | Granularity | Coverage | Value | +|-----------------|---------------|---------------|-----------|-------| +| {metric1} | {description} | {daily/weekly}| {geo} | ā­ā­ā­ā­ā­ | +| {metric2} | {description} | {monthly} | {geo} | ā­ā­ā­ā­ā­ | +| {metric3} | {description} | {annual} | {geo} | ā­ā­ā­ā­ | +... + +**Real Example (NASS):** + +| Metric Type | Data | Frequency | Value | Implement? | +|----------------|--------------------| ----------|----------|------------| +| CONDITION | Quality ratings | Weekly | ā­ā­ā­ā­ā­ | āœ… YES | +| PROGRESS | % planted/harvested| Weekly | ā­ā­ā­ā­ā­ | āœ… YES | +| YIELD | Bu/acre | Monthly | ā­ā­ā­ā­ā­ | āœ… YES | +| PRODUCTION | Total bushels | Monthly | ā­ā­ā­ā­ā­ | āœ… YES | +| AREA | Acres planted | Annual | ā­ā­ā­ā­ | āœ… YES | +| PRICE | $/bushel | Monthly | ā­ā­ā­ | āšŖ v2.0 | +``` + +**Step 4.2: Coverage Decision** + +**GOLDEN RULE:** +- If metric has ā­ā­ā­ā­ or ā­ā­ā­ā­ā­ value ā†’ Implement in v1.0 +- If API has 5 high-value metrics ā†’ Implement all 5! +- Never leave >50% of API unused without strong justification + +**Step 4.3: Document Decision** + +In DECISIONS.md: +```markdown +## API Coverage Decision + +API {name} offers {N} types of metrics. + +**Implemented in v1.0 ({X} of {N}):** +- {metric1} - {justification} +- {metric2} - {justification} +... + +**Not implemented ({Y} of {N}):** +- {metricZ} - {why not} (planned for v2.0) + +**Coverage:** {X/N * 100}% = {evaluation} +- If < 70%: Clearly explain why low coverage +- If > 70%: āœ… Good coverage +``` + +**Output of this phase:** Exact list of all `get_*()` methods to implement +- Examples: [many/few/none] +- SDKs: [Python/R/None] + +**Ease of Use**: +- Format: JSON / CSV / XML +- Structure: [simple/complex] +- Quirks: [any strange behavior?] +``` + +### Step 4: Compare Options + +Create comparison table: + +| API | Coverage | Cost | Rate Limit | Quality | Docs | Ease | Score | +|-----|-----------|-------|------------|-----------|------|------------|-------| +| API 1 | ā­ā­ā­ā­ā­ | Free | 1000/day | Official | ā­ā­ā­ā­ | ā­ā­ā­ā­ā­ | 9.2/10 | +| API 2 | ā­ā­ā­ā­ | $49/mo | Unlimited | Private | ā­ā­ā­ā­ā­ | ā­ā­ā­ā­ | 7.8/10 | +| API 3 | ā­ā­ā­ | Free | 100/day | Private | ā­ā­ | ā­ā­ā­ | 5.5/10 | + +**Scoring criteria**: +- Coverage (fit with need): 30% weight +- Cost (prefer free): 20% weight +- Rate limit (sufficient?): 15% weight +- Quality (official > private): 15% weight +- Documentation (facilitates implementation): 10% weight +- Ease of use (format, structure): 10% weight + +### Step 5: DECIDE + +**Consider user constraints**: +- Mentioned "free"? ā†’ Eliminate paid options +- Mentioned "10+ years historical data"? ā†’ Check coverage +- Mentioned "real-time"? ā†’ Prioritize streaming APIs + +**Apply logic**: +1. Eliminate APIs that violate constraints +2. Of remaining, choose highest score +3. If tie, prefer: + - Official > private + - Better documentation + - Easier to use + +**FINAL DECISION**: + +```markdown +## Selected API: [API Name] + +**Score**: X.X/10 + +**Justification**: +- āœ… Coverage: [specific details] +- āœ… Cost: [free/paid + details] +- āœ… Rate limit: [number] requests/day (sufficient for [estimated usage]) +- āœ… Quality: [official/private + reliability] +- āœ… Documentation: [quality + examples] +- āœ… Ease of use: [format, structure] + +**Fit with requirements**: +- Constraint 1 (e.g., free): āœ… Met +- Constraint 2 (e.g., 10+ years history): āœ… Met (since [year]) +- Primary need (e.g., crop production): āœ… Covered + +**Alternatives Considered**: + +**API X**: Score 7.5/10 +- Rejected because: [specific reason] +- Trade-off: [what we lose vs gain] + +**API Y**: Score 6.2/10 +- Rejected because: [reason] + +**Conclusion**: +[API Name] is the best option because [1-2 sentence synthesis]. +``` + +### Step 6: Research Technical Details + +After deciding, dive deep into documentation: + +**Load via WebFetch**: +- Getting started guide +- Complete API reference +- Authentication guide +- Rate limiting details +- Best practices + +**Extract for implementation**: + +```markdown +## Technical Details - [API] + +### Authentication + +**Method**: API key in header +**Header**: `X-Api-Key: YOUR_KEY` +**Obtaining key**: +1. [step 1] +2. [step 2] +3. [step 3] + +### Main Endpoints + +**Endpoint 1**: [Name] +- **URL**: `GET https://api.example.com/v1/endpoint` +- **Parameters**: + - `param1` (required): [description, type, example] + - `param2` (optional): [description, type, default] +- **Response** (200 OK): +```json +{ + "data": [...], + "meta": {...} +} +``` +- **Errors**: + - 400: [when occurs, how to handle] + - 401: [when occurs, how to handle] + - 429: [rate limit, how to handle] + +**Example request**: +```bash +curl -H "X-Api-Key: YOUR_KEY" \ + "https://api.example.com/v1/endpoint?param1=value" +``` + +[Repeat for all relevant endpoints] + +### Rate Limiting + +- Limit: [number] requests per [period] +- Response headers: + - `X-RateLimit-Limit`: Total limit + - `X-RateLimit-Remaining`: Remaining requests + - `X-RateLimit-Reset`: Reset timestamp +- Behavior when exceeded: [429 error, throttling, ban?] +- Best practice: [how to implement rate limiting] + +### Quirks and Gotchas + +**Quirk 1**: Values come as strings with formatting +- Example: `"2,525,000"` instead of `2525000` +- Solution: Remove commas before converting + +**Quirk 2**: Suppressed data marked as "(D)" +- Meaning: Withheld to avoid disclosing data +- Solution: Treat as NULL, signal to user + +**Quirk 3**: [other non-obvious behavior] +- Solution: [how to handle] + +### Performance Tips + +- Historical data doesn't change ā†’ cache permanently +- Recent data may be revised ā†’ short cache (7 days) +- Use pagination parameters if large response +- Make parallel requests when possible (respecting rate limit) +``` + +### Step 7: Document for Later Use + +Save everything in `references/api-guide.md` of the agent to be created. + +## Discovery Examples + +### Example 1: US Agriculture + +**Input**: "US crop data" + +**Research**: +``` +WebSearch: "USDA API agricultural data" +ā†’ Found: NASS QuickStats, ERS, FAS + +WebFetch: https://quickstats.nass.usda.gov/api +ā†’ Free, data since 1866, 1000/day rate limit + +WebFetch: https://www.ers.usda.gov/developer/ +ā†’ Free, economic focus, less granular + +WebFetch: https://apps.fas.usda.gov/api +ā†’ International focus, not domestic +``` + +**Comparison**: +| API | Coverage (US domestic) | Cost | Production Data | Score | +|-----|---------------------------|-------|-------------------|-------| +| NASS | ā­ā­ā­ā­ā­ (excellent) | Free | ā­ā­ā­ā­ā­ | 9.5/10 | +| ERS | ā­ā­ā­ā­ (good) | Free | ā­ā­ā­ (economic) | 7.0/10 | +| FAS | ā­ā­ (international) | Free | ā­ā­ (global) | 4.0/10 | + +**DECISION**: NASS QuickStats API +- Best coverage for US domestic agriculture +- Free with reasonable rate limit +- Complete production, area, yield data + +### Example 2: Stock Market + +**Input**: "technical stock analysis" + +**Research**: +``` +WebSearch: "stock market API free historical data" +ā†’ Alpha Vantage, Yahoo Finance, IEX Cloud, Polygon.io + +WebFetch: Alpha Vantage docs +ā†’ Free, 5 requests/min, 500/day + +WebFetch: Yahoo Finance (yfinance) +ā†’ Free, unlimited but unofficial + +WebFetch: IEX Cloud +ā†’ Freemium, good docs, 50k free credits/month +``` + +**Comparison**: +| API | Data | Cost | Rate Limit | Official | Score | +|-----|-------|-------|------------|---------|-------| +| Alpha Vantage | Complete | Free | 500/day | ā­ā­ā­ | 8.0/10 | +| Yahoo Finance | Complete | Free | Unlimited | āŒ Unofficial | 7.5/10 | +| IEX Cloud | Excellent | Freemium | 50k/month | ā­ā­ā­ā­ | 8.5/10 | + +**DECISION**: IEX Cloud (free tier) +- Official and reliable +- 50k requests/month sufficient +- Excellent documentation +- Complete data (OHLCV + volume) + +### Example 3: Global Climate + +**Input**: "global climate data" + +**Research**: +``` +WebSearch: "weather API historical data global" +ā†’ NOAA, OpenWeather, Weather.gov, Meteostat + +[Research each one...] +``` + +**DECISION**: NOAA Climate Data Online (CDO) API +- Official (US government) +- Free +- Global and historical coverage (1900+) +- Rate limit: 1000/day + +## Decision Documentation + +Create `DECISIONS.md` file in agent: + +```markdown +# Architecture Decisions + +## Date: [creation date] + +## Phase 1: API Selection + +### Chosen API + +**[API Name]** + +### Selection Process + +**APIs Researched**: [list] + +**Evaluation Criteria**: +1. Data coverage (fit with need) +2. Cost (preference for free) +3. Rate limits (viability) +4. Quality (official > private) +5. Documentation (facilitates development) + +### Comparison + +[Comparison table] + +### Final Justification + +[2-3 paragraphs explaining why this API was chosen] + +### Trade-offs + +**What we gain**: +- [benefit 1] +- [benefit 2] + +**What we lose** (vs alternatives): +- [accepted limitation 1] +- [accepted limitation 2] + +### Technical Details + +[Summary of endpoints, authentication, rate limits, etc] + +**Complete documentation**: See `references/api-guide.md` +``` + +## Phase 1 Checklist + +Before proceeding to Phase 2, verify: + +- [ ] Research completed (WebSearch + WebFetch) +- [ ] Minimum 3 APIs compared +- [ ] Decision made with clear justification +- [ ] User constraints respected +- [ ] Technical details extracted +- [ ] DECISIONS.md created +- [ ] Ready for analysis design diff --git a/references/phase2-design.md b/references/phase2-design.md new file mode 100644 index 0000000..ddcd837 --- /dev/null +++ b/references/phase2-design.md @@ -0,0 +1,244 @@ +# Phase 2: Analysis Design + +## Objective + +**DEFINE** autonomously which analyses the agent will perform and how. + +## Detailed Process + +### Step 1: Brainstorm Use Cases + +From the workflow described by the user, think of typical questions they will ask. + +**Technique**: "If I were this user, what would I ask?" + +**Example (US agriculture)**: + +User said: "download crop data, compare year vs year, make rankings" + +**Questions they likely ask**: +1. "What's the corn production in 2023?" +2. "How's soybean compared to last year?" +3. "Did production grow or fall?" +4. "How much did it grow?" +5. "Does growth come from area or productivity?" +6. "Which states produce most wheat?" +7. "Top 5 soybean producers" +8. "Did the ranking change vs last year?" +9. "Production trend last 5 years?" +10. "Forecast for next year?" +11. "Average US yield" +12. "Which state has best productivity?" +13. "Did planted area increase?" +14. "Compare Midwest vs South" +15. "Production by region" + +**Objective**: List 15-20 typical questions + +### Step 2: Group by Analysis Type + +Group similar questions: + +**Group 1: Simple Queries** (fetching + formatting) +- Questions: 1, 11, 13 +- Required analysis: **Data Retrieval** +- Complexity: Low + +**Group 2: Temporal Comparisons** (YoY) +- Questions: 2, 3, 4, 5 +- Required analysis: **YoY Comparison + Decomposition** +- Complexity: Medium + +**Group 3: Rankings** (sorting + share) +- Questions: 6, 7, 8 +- Required analysis: **State Ranking** +- Complexity: Medium + +**Group 4: Trends** (time series) +- Questions: 9 +- Required analysis: **Trend Analysis** +- Complexity: Medium-High + +**Group 5: Projections** (forecasting) +- Questions: 10 +- Required analysis: **Forecasting** +- Complexity: High + +**Group 6: Geographic Aggregations** +- Questions: 12, 14, 15 +- Required analysis: **Regional Aggregation** +- Complexity: Medium + +### Step 3: Prioritize Analyses + +**Prioritization criteria**: +1. **Frequency of use** (based on described workflow) +2. **Analytical value** (insight vs effort) +3. **Implementation complexity** (easier first) +4. **Dependencies** (does one analysis depend on another?) + +**Scoring**: + +| Analysis | Frequency | Value | Ease | Score | +|---------|------------|-------|------------|-------| +| YoY Comparison | ā­ā­ā­ā­ā­ | ā­ā­ā­ā­ā­ | ā­ā­ā­ā­ | 9.3/10 | +| State Ranking | ā­ā­ā­ā­ā­ | ā­ā­ā­ā­ | ā­ā­ā­ā­ā­ | 9.3/10 | +| Regional Agg | ā­ā­ā­ā­ | ā­ā­ā­ā­ | ā­ā­ā­ā­ | 8.0/10 | +| Trend Analysis | ā­ā­ā­ | ā­ā­ā­ā­ā­ | ā­ā­ā­ | 7.3/10 | +| Data Retrieval | ā­ā­ā­ā­ā­ | ā­ā­ā­ | ā­ā­ā­ā­ā­ | 8.3/10 | +| Forecasting | ā­ā­ | ā­ā­ā­ā­ | ā­ā­ | 5.3/10 | + +**DECISION**: Implement top 5 +1. YoY Comparison (9.3) +2. State Ranking (9.3) +3. Data Retrieval (8.3) +4. Regional Aggregation (8.0) +5. Trend Analysis (7.3) + +**Don't implement initially** (can add later): +- Forecasting (5.3) - complex, occasional use + +### Step 4: Specify Each Analysis + +For each selected analysis: + +```markdown +## Analysis: [Name] + +**Objective**: [What it does in 1 sentence] + +**When to use**: [Types of questions that trigger it] + +**Required inputs**: +- Input 1: [type, description] +- Input 2: [type, description] + +**Expected outputs**: +- Output 1: [type, description] +- Output 2: [type, description] + +**Methodology**: + +[Explanation in natural language] + +**Formulas**: +``` +Formula 1 = ... +Formula 2 = ... +``` + +**Data transformations**: +1. [Transformation 1] +2. [Transformation 2] + +**Validations**: +- Validation 1: [criteria] +- Validation 2: [criteria] + +**Interpretation**: +- If result > X: [interpretation] +- If result < Y: [interpretation] + +**Concrete example**: + +Input: +- Commodity: Corn +- Year current: 2023 +- Year previous: 2022 + +Processing: +- Fetch 2023 production: 15.3B bu +- Fetch 2022 production: 13.7B bu +- Calculate: (15.3 - 13.7) / 13.7 = +11.7% + +Output: +```json +{ + "commodity": "CORN", + "year_current": 2023, + "year_previous": 2022, + "production_current": 15.3, + "production_previous": 13.7, + "change_absolute": 1.6, + "change_percent": 11.7, + "interpretation": "significant_increase" +} +``` + +Response to user: +"Corn production grew 11.7% in 2023 vs 2022 (15.3B bu vs 13.7B bu)." +``` + +### Step 5: Specify Methodologies + +For quantitative analyses, detail methodology: + +**Example: YoY Decomposition** + +```markdown +### Growth Decomposition + +**Objective**: Understand how much of production growth comes from: +- Planted area increase (extensive) +- Productivity/yield increase (intensive) + +**Mathematics**: + +Production = Area Ɨ Yield + +Ī” Production = Ī” Area Ɨ Yield(t-1) + Area(t-1) Ɨ Ī” Yield + Ī” Area Ɨ Ī” Yield + +Interaction term usually small, so approximation: + +Ī” Production ā‰ˆ Ī” Area Ɨ Yield(t-1) + Area(t-1) Ɨ Ī” Yield + +**Percentage contributions**: + +Contrib_Area = (Ī” Area% / Ī” Production%) Ɨ 100 +Contrib_Yield = (Ī” Yield% / Ī” Production%) Ɨ 100 + +**Interpretation**: + +- Contrib_Area > 60%: **Extensive growth** + ā†’ Area expansion is main driver + ā†’ Agricultural frontier expanding + +- Contrib_Yield > 60%: **Intensive growth** + ā†’ Technology improvement is main driver + ā†’ Productivity/ha increasing + +- Both ~50%: **Balanced growth** + +**Validation**: + +Check: Production(t) ā‰ˆ Area(t) Ɨ Yield(t) (margin 1%) +Check: Contrib_Area + Contrib_Yield ā‰ˆ 100% (margin 5%) + +**Example**: + +Soybeans 2023 vs 2022: +- Ī” Production: +12.4% +- Ī” Area: +6.1% +- Ī” Yield: +7.6% + +Contrib_Area = (6.1 / 12.4) Ɨ 100 = 49% +Contrib_Yield = (7.6 / 12.4) Ɨ 100 = 61% + +**Interpretation**: Intensive growth (61% from yield). +Technology and management improving. +``` + +### Step 6: Document Analyses + +Save all specifications in `references/analysis-methods.md` of the agent. + +## Phase 2 Checklist + +- [ ] 15+ typical questions listed +- [ ] Questions grouped by analysis type +- [ ] 4-6 analyses prioritized (with scoring) +- [ ] Each analysis specified (objective, inputs, outputs, methodology) +- [ ] Methodologies detailed with formulas +- [ ] Validations defined +- [ ] Interpretations specified +- [ ] Concrete examples included diff --git a/references/phase3-architecture.md b/references/phase3-architecture.md new file mode 100644 index 0000000..23d107c --- /dev/null +++ b/references/phase3-architecture.md @@ -0,0 +1,351 @@ +# Phase 3: Architecture and Structuring + +## Objective + +**STRUCTURE** the agent optimally: folders, files, responsibilities, cache, performance. + +## Detailed Process + +### Step 1: Define Agent Name + +Based on domain and objective, create descriptive name: + +**Format**: `domain-objective-type` + +**Examples**: +- US Agriculture ā†’ `nass-usda-agriculture` +- Stock analysis ā†’ `stock-technical-analysis` +- Global climate ā†’ `noaa-climate-analysis` +- Brazil CONAB ā†’ `conab-crop-yield-analysis` + +**Rules**: +- lowercase +- hyphens to separate words +- maximum 50 characters +- descriptive but concise + +### Step 2: Directory Structure + +**Decision**: How many levels of organization? + +**Option A - Simple** (small agents): +``` +agent-name/ +ā”œā”€ā”€ .claude-plugin/ +ā”‚ ā””ā”€ā”€ marketplace.json ā† REQUIRED for installation! +ā”œā”€ā”€ SKILL.md +ā”œā”€ā”€ scripts/ +ā”‚ ā””ā”€ā”€ main.py +ā”œā”€ā”€ references/ +ā”‚ ā””ā”€ā”€ guide.md +ā””ā”€ā”€ assets/ + ā””ā”€ā”€ config.json +``` + +**Option B - Organized** (medium agents): +``` +agent-name/ +ā”œā”€ā”€ .claude-plugin/ +ā”‚ ā””ā”€ā”€ marketplace.json ā† REQUIRED for installation! +ā”œā”€ā”€ SKILL.md +ā”œā”€ā”€ scripts/ +ā”‚ ā”œā”€ā”€ fetch.py +ā”‚ ā”œā”€ā”€ parse.py +ā”‚ ā”œā”€ā”€ analyze.py +ā”‚ ā””ā”€ā”€ utils/ +ā”‚ ā”œā”€ā”€ cache.py +ā”‚ ā””ā”€ā”€ validators.py +ā”œā”€ā”€ references/ +ā”‚ ā”œā”€ā”€ api-guide.md +ā”‚ ā””ā”€ā”€ methodology.md +ā””ā”€ā”€ assets/ + ā””ā”€ā”€ config.json +``` + +**Option C - Complete** (complex agents): +``` +agent-name/ +ā”œā”€ā”€ .claude-plugin/ +ā”‚ ā””ā”€ā”€ marketplace.json ā† REQUIRED for installation! +ā”œā”€ā”€ SKILL.md +ā”œā”€ā”€ scripts/ +ā”‚ ā”œā”€ā”€ core/ +ā”‚ ā”‚ ā”œā”€ā”€ fetch_[source].py +ā”‚ ā”‚ ā”œā”€ā”€ parse_[source].py +ā”‚ ā”‚ ā””ā”€ā”€ analyze_[source].py +ā”‚ ā”œā”€ā”€ models/ +ā”‚ ā”‚ ā”œā”€ā”€ forecasting.py +ā”‚ ā”‚ ā””ā”€ā”€ ml_models.py +ā”‚ ā””ā”€ā”€ utils/ +ā”‚ ā”œā”€ā”€ cache_manager.py +ā”‚ ā”œā”€ā”€ rate_limiter.py +ā”‚ ā””ā”€ā”€ validators.py +ā”œā”€ā”€ references/ +ā”‚ ā”œā”€ā”€ api/ +ā”‚ ā”‚ ā””ā”€ā”€ [api-name]-guide.md +ā”‚ ā”œā”€ā”€ methods/ +ā”‚ ā”‚ ā””ā”€ā”€ analysis-methods.md +ā”‚ ā””ā”€ā”€ troubleshooting.md +ā”œā”€ā”€ assets/ +ā”‚ ā”œā”€ā”€ config.json +ā”‚ ā””ā”€ā”€ metadata.json +ā””ā”€ā”€ data/ + ā”œā”€ā”€ raw/ + ā”œā”€ā”€ processed/ + ā”œā”€ā”€ cache/ + ā””ā”€ā”€ analysis/ +``` + +**Choose based on**: +- Number of scripts (1-2 ā†’ A, 3-5 ā†’ B, 6+ ā†’ C) +- Analysis complexity +- Prefer starting with B, expand to C if needed + +### Step 3: Define Script Responsibilities + +**Principle**: Separation of Concerns + +**Typical scripts**: + +**1. fetch_[source].py** +- **Responsibility**: API requests, authentication, rate limiting +- **Input**: Query parameters (commodity, year, etc) +- **Output**: Raw JSON from API +- **Does NOT**: Parse, transform, analyze +- **Size**: 200-300 lines + +**2. parse_[source].py** +- **Responsibility**: Parsing, cleaning, validation +- **Input**: API JSON +- **Output**: Structured DataFrame +- **Does NOT**: Fetch, analyze +- **Size**: 150-200 lines + +**3. analyze_[source].py** +- **Responsibility**: All analyses (YoY, ranking, etc) +- **Input**: Clean DataFrame +- **Output**: Dict with results +- **Does NOT**: Fetch, parse +- **Size**: 300-500 lines (all analyses) + +**Typical utils**: + +**cache_manager.py**: +- Manage response cache +- Differentiated TTL +- ~100-150 lines + +**rate_limiter.py**: +- Control rate limit +- Persistent counter +- Alerts +- ~100-150 lines + +**validators.py**: +- Data validations +- Consistency checks +- ~100-150 lines + +**unit_converter.py** (if needed): +- Unit conversions +- ~50-100 lines + +### Step 4: Plan References + +**Typical files**: + +**api-guide.md** (~1500 words): +- How to get API key +- Main endpoints with examples +- Important parameters +- Response format +- Limitations and quirks +- API troubleshooting + +**analysis-methods.md** (~2000 words): +- Each analysis explained +- Mathematical formulas +- Interpretations +- Validations +- Concrete examples + +**troubleshooting.md** (~1000 words): +- Common problems +- Step-by-step solutions +- FAQs + +**domain-context.md** (optional, ~1000 words): +- Domain context +- Terminology +- Important concepts +- Benchmarks + +### Step 5: Plan Assets + +**config.json**: +- API settings (URL, rate limits, timeouts) +- Cache settings (TTLs, directories) +- Analysis defaults +- Validation thresholds + +**Typical structure**: +```json +{ + "api": { + "base_url": "...", + "api_key_env": "VAR_NAME", + "rate_limit_per_day": 1000, + "timeout_seconds": 30 + }, + "cache": { + "enabled": true, + "dir": "data/cache", + "ttl_historical_days": 365, + "ttl_current_days": 7 + }, + "defaults": { + "param1": "value1" + }, + "validation": { + "threshold1": 0.01 + } +} +``` + +**metadata.json** (if needed): +- Domain-specific metadata +- Mappings (aliases) +- Conversions (units) +- Groupings (regions) + +**Example**: +```json +{ + "commodities": { + "CORN": { + "aliases": ["corn", "maize"], + "unit": "BU", + "conversion_to_mt": 0.0254 + } + }, + "regions": { + "MIDWEST": ["IA", "IL", "IN", "OH"] + } +} +``` + +### Step 6: Cache Strategy + +**Decision**: What to cache and for how long? + +**General rule**: +- **Historical data** (year < current): Permanent cache (365+ days) +- **Current year data**: Short cache (7 days - may be revised) +- **Metadata** (commodity lists, states): Permanent cache + +**Implementation**: +- Cache by key (parameter hash) +- Check age before using +- Fallback to expired cache if API fails + +**Example**: +```python +def get_cache_ttl(year: int) -> timedelta: + """Determine cache TTL based on year""" + current_year = datetime.now().year + + if year < current_year: + # Historical: cache for 1 year (effectively permanent) + return timedelta(days=365) + else: + # Current year: cache for 7 days (may be revised) + return timedelta(days=7) +``` + +### Step 7: Rate Limiting Strategy + +**Decision**: How to control rate limits? + +**Components**: +1. **Persistent counter** (file/DB) +2. **Pre-request verification** +3. **Alerts** (when near limit) +4. **Blocking** (when limit reached) + +**Implementation**: +```python +class RateLimiter: + def __init__(self, max_requests: int, period_seconds: int): + self.max_requests = max_requests + self.period = period_seconds + self.counter_file = Path("data/cache/rate_limit_counter.json") + + def allow_request(self) -> bool: + """Check if request is allowed""" + count = self._get_current_count() + + if count >= self.max_requests: + return False + + # Warn when near limit + if count > self.max_requests * 0.9: + print(f"āš ļø Rate limit: {count}/{self.max_requests} requests used") + + return True + + def record_request(self): + """Record that request was made""" + count = self._get_current_count() + self._save_count(count + 1) +``` + +### Step 8: Document Architecture + +Create section in DECISIONS.md: + +```markdown +## Phase 3: Architecture + +### Chosen Structure + +``` +[Directory tree] +``` + +**Justification**: +- Separate scripts for modularity +- Utils for reusable code +- References for progressive disclosure +- Data/ to separate raw vs processed + +### Defined Scripts + +**fetch_[source].py** (280 lines estimated): +- Responsibility: [...] +- Input/Output: [...] + +[For each script...] + +### Cache Strategy + +- Historical: Permanent cache +- Current: 7 day cache +- Justification: [historical data doesn't change] + +### Rate Limiting + +- Method: [persistent counter] +- Limits: [1000/day] +- Alerts: [>90% usage] +``` + +## Phase 3 Checklist + +- [ ] Agent name defined +- [ ] Directory structure chosen (A/B/C) +- [ ] Responsibilities of each script defined +- [ ] References planned (which files, content) +- [ ] Assets planned (which configs, structure) +- [ ] Cache strategy defined (what, TTL) +- [ ] Rate limiting strategy defined +- [ ] Architecture documented in DECISIONS.md diff --git a/references/phase4-detection.md b/references/phase4-detection.md new file mode 100644 index 0000000..79a2706 --- /dev/null +++ b/references/phase4-detection.md @@ -0,0 +1,465 @@ +# Phase 4: Automatic Detection + +## Objective + +**DETERMINE** keywords and create description so Claude Code activates the skill automatically. + +## Detailed Process + +### Step 1: List Domain Entities + +Identify all relevant entities that users may mention: + +**Entity categories**: + +**1. Organizations/Sources** +- Organization names (USDA, CONAB, NOAA, IMF) +- Acronyms (NASS, ERS, FAS) +- Full names (National Agricultural Statistics Service) + +**2. Main Objects** +- For agriculture: commodities (corn, soybeans, wheat) +- For finance: instruments (stocks, bonds, options) +- For climate: metrics (temperature, precipitation) + +**3. Geography** +- Countries (US, Brazil, China) +- Regions (Midwest, Centro-Oeste, Southeast) +- States/Provinces (Iowa, Mato Grosso, Texas) + +**4. Metrics** +- Production, area, yield, price +- Revenue, profit, growth +- Temperature, rainfall, humidity + +**5. Temporality** +- Years, seasons, quarters, months +- Current, historical, forecast +- YoY, QoQ, MoM + +**Example (US agriculture)**: + +```markdown +**Organizations**: +- USDA, NASS, National Agricultural Statistics Service +- Department of Agriculture +- QuickStats + +**Commodities**: +- Corn, soybeans, wheat +- Cotton, rice, sorghum +- Barley, oats, hay, peanuts +- [list all major ones - 20+] + +**Geography**: +- US, United States, national +- States: Iowa, Illinois, Nebraska, Kansas, Texas, etc [list top 15] +- Regions: Midwest, Great Plains, Southeast, etc + +**Metrics**: +- Production, area planted, area harvested +- Yield, productivity +- Price received, value of production +- Inventory, stocks + +**Temporality**: +- Year, season, crop year +- Current, latest, this year, last year +- Historical, trend, past 5 years +- Forecast, projection, outlook +``` + +### Step 2: List Actions/Verbs + +Which verbs does the user use to request analyses? + +**Categories**: + +**Query (fetch information)**: +- What is, how much, show me, get +- Tell me, find, retrieve + +**Compare**: +- Compare, versus, vs, against +- Difference, change, growth +- Higher, lower, better, worse + +**Rank (sort)**: +- Top, best, leading, biggest +- Rank, ranking, list +- Which states, which countries + +**Analyze**: +- Analyze, analysis +- Trend, pattern, evolution +- Breakdown, decompose, explain + +**Forecast (project)**: +- Predict, project, forecast +- Outlook, expectation, estimate +- Future, next year, coming season + +**Visualize**: +- Plot, chart, graph, visualize +- Show chart, generate graph + +### Step 2.5: Generate Exhaustive Keywords (NEW v2.0 - CRITICAL!) + +**OBJECTIVE:** Generate 60+ keywords to ensure correct activation in ALL relevant queries. + +**LEARNING:** us-crop-monitor v1.0 had ~20 keywords. Missing "yield", "harvest", "production" ā†’ Claude Code didn't activate for those queries. v2.0 expanded to 60+ keywords. + +**Mandatory Process:** + +**Step A: Keywords per API Metric** + +For EACH metric/endpoint the skill implements, generate keywords: + +```markdown +Metric 1: CONDITION (quality ratings) +Primary keywords: condition, conditions, quality, ratings +Secondary keywords: status, health, state +Technical keywords: excellent, good, fair, poor +Action keywords: rate, rated, rating, classify +Portuguese: condiĆ§Ć£o, condiƧƵes, qualidade, estado, classificaĆ§Ć£o +ā†’ Total: ~15 keywords + +Metric 2: PROGRESS (% planted/harvested) +Primary keywords: progress, harvest, planted, harvested +Secondary keywords: planting, harvesting, completion +Technical keywords: percentage, percent, % +Action keywords: advancing, complete, completed +Portuguese: progresso, plantio, colheita, plantado, colhido +ā†’ Total: ~15 keywords + +Metric 3: YIELD (productivity) +Primary keywords: yield, productivity, performance +Technical keywords: bushels per acre, bu/acre, bu/ac +Secondary keywords: output per unit +Portuguese: rendimento, produtividade, bushels por acre +ā†’ Total: ~12 keywords + +... Repeat for ALL implemented metrics +``` + +**Rule:** Each metric = minimum 10 unique keywords + +**Step B: Categorize Keywords by Type** + +```markdown +### Keyword Matrix - {Skill Name} + +**1. Main Entities** (20+ keywords) +- Official name: {entity} +- Variations: {variations} +- Singular + plural +- Acronyms: {acronyms} +- Full names: {full names} +- Portuguese: {portuguese terms} + +**2. Metrics - ONE SECTION PER API METRIC!** (30+ keywords) +- Metric 1: {list 10-15 keywords} +- Metric 2: {list 10-15 keywords} +- Metric 3: {list 10-15 keywords} +... + +**3. Actions/Verbs** (20+ keywords) +- Query: what, how, show, get, tell, find, retrieve +- Compare: compare, vs, versus, against, difference +- Rank: top, best, rank, leading, biggest +- Analyze: analyze, trend, pattern, evolution +- Report: report, dashboard, summary, overview +- Portuguese: comparar, ranking, anĆ”lise, relatĆ³rio + +**4. Temporal Qualifiers** (15+ keywords) +- Current: current, now, today, latest, recent, atual, agora, hoje +- Historical: historical, past, previous, last year, histĆ³rico +- Comparative: this year vs last year, YoY, year-over-year +- Forecast: forecast, projection, estimate, outlook, previsĆ£o + +**5. Geographic Qualifiers** (15+ keywords) +- National: national, US, United States, country-wide +- Regional: region, Midwest, South, regional +- State: state, by state, state-level, estado +- Specific names: Iowa, Illinois, Nebraska, ... + +**6. Data Context** (10+ keywords) +- Source: {API name}, {organization}, {data source} +- Type: data, statistics, metrics, indicators, dados +``` + +**Goal:** Total 60-80 unique keywords! + +**Step C: Test Coverage Matrix** + +For each analysis function, generate 10 different queries: + +```markdown +Function: harvest_progress_report() + +Query variations (test coverage): +1. "What's the corn harvest progress?" āœ… harvest, progress +2. "How much corn has been harvested?" āœ… harvested +3. "Percent corn harvested?" āœ… percent, harvested +4. "Harvest completion status?" āœ… harvest, completion, status +5. "Progresso de colheita do milho?" āœ… progresso, colheita +6. "Quanto foi colhido?" āœ… colhido +7. "Harvest advancement?" āœ… harvest, advancement +8. "How advanced is harvest?" āœ… harvest, advanced +9. "Colheita completa?" āœ… colheita +10. "Percentage complete harvest?" āœ… percentage, harvest + +ALL keywords present in description? ā†’ Verify! +``` + +**Do this for ALL 11 functions** = 110 query variations tested! + +### Step 3: List Question Variations + +For each analysis type, how can user ask? + +**YoY Comparison**: +- "Compare X this year vs last year" +- "How does X compare to last year" +- "Is X up or down from last year" +- "X growth rate" +- "X change YoY" +- "X vs previous year" +- "Did X increase or decrease" + +**Ranking**: +- "Top states for X" +- "Which states produce most X" +- "Leading X producers" +- "Best X production" +- "Biggest X producers" +- "Ranking of X" +- "List top 10 X" + +**Trend**: +- "X trend last N years" +- "How has X changed over time" +- "X evolution" +- "Historical X data" +- "X growth rate historical" +- "Long term trend of X" + +**Simple Query**: +- "What is X production" +- "X production in [year]" +- "How much X" +- "X data" +- "Current X" + +### Step 4: Define Negative Scope + +**Important**: What should NOT activate? + +Avoid false positives (skill activates when it shouldn't). + +**Technique**: Think of similar questions but OUT of scope. + +**Example (US agriculture)**: + +āŒ **DO NOT activate for**: +- Futures market prices + - "CBOT corn futures price" + - "Soybean futures December contract" + - Reason: Skill is USDA data (physical production), not trading + +- Other countries' agriculture + - "Brazil soybean production" + - "Argentina corn exports" + - Reason: Skill is US only + +- Consumption/demand + - "US corn consumption" + - "Soybean demand forecast" + - Reason: NASS has production, not consumption + +- Private company data + - "Monsanto corn seed sales" + - "Cargill soybean crush" + - Reason: Corporate data, not national statistics + +**Document**: +```markdown +## Skill Scope + +### āœ… WITHIN scope: +- Physical crop production in US +- Planted/harvested area +- Yield/productivity +- Prices RECEIVED by farmers (farm gate) +- Inventories +- Historical and current data +- Comparisons, rankings, trends + +### āŒ OUT of scope: +- Futures market prices (CBOT, CME) +- Agriculture outside US +- Consumption/demand +- Private company data +- Market price forecasting +``` + +### Step 5: Create Precise Description (Updated v2.0) + +**NEW RULE:** Description must contain ALL 60+ identified keywords! + +**Expanded Template:** + +```yaml +description: This skill should be used when the user asks about +{domain} ({main entities with variations}). Automatically activates +for queries about {metric1} ({metric1 keywords}), {metric2} +({metric2 keywords}), {metric3} ({metric3 keywords}), {metric4} +({metric4 keywords}), {metric5} ({metric5 keywords}), {actions_list}, +{temporal qualifiers}, {geographic qualifiers}, comparisons +{comparison types}, rankings, trends, {data source} data, +comprehensive reports, and dashboards. Uses {language} with {API name} +to fetch real data on {complete list of all metrics}. +``` + +**Mandatory components**: +1. āœ… **Domain** with entities (corn, soybeans, wheat - not just "crops") +2. āœ… **EACH API metric** explicitly mentioned +3. āœ… **Synonyms** in parentheses (harvest = colheita, yield = rendimento) +4. āœ… **Actions** covered (compare, rank, analyze, report) +5. āœ… **Temporal context** (current, today, year-over-year) +6. āœ… **Geographic** context (states, regions, national) +7. āœ… **Data source** (USDA NASS, etc.) +8. āœ… **Portuguese + English** keywords mixed + +**Real size:** 300-500 characters (yes, larger than "recommended" - but necessary!) + +**Real Example (us-crop-monitor v2.0):** +```yaml +description: This skill should be used when the user asks about +agricultural crops in the United States (soybeans, corn, wheat). +Automatically activates for queries about crop conditions (condiƧƵes), +crop progress (progresso de plantio/colheita), harvest progress +(progresso de colheita), planting progress (plantio), yield +(produtividade/rendimento em bushels per acre), production (produĆ§Ć£o +total em bushels), area planted (Ć”rea plantada), area harvested +(Ć”rea colhida), acres, forecasts (estimativas), crop monitoring, +weekly comparisons (week-over-week) or annual (year-over-year), +state producer rankings, trend analyses, USDA NASS data, comprehensive +reports, and crop dashboards. Uses Python with NASS API to fetch +real data on condition, progress, productivity, production and area. +``` + +**Analysis:** +- Entities: soybeans, corn, wheat (3) +- Metrics: conditions, progress, harvest, planting, yield, production, area (7) +- Each metric with PT synonym: (condiƧƵes), (colheita), (rendimento), etc. +- Actions: queries, comparisons, rankings, analyses, reports +- Temporal: weekly, annual, week-over-week, year-over-year +- Source: USDA NASS +- Total unique keywords: ~65+ + +**Step D: Validate Keyword Coverage** + +Final checklist: +```markdown +- [ ] All API metrics mentioned? (if API has 5 ā†’ 5 in description) +- [ ] Each metric has PT synonym? (yield = rendimento) +- [ ] Action verbs included? (compare, rank, analyze) +- [ ] Temporal context? (current, today, YoY) +- [ ] Geographic context? (states, national) +- [ ] Data source mentioned? (USDA NASS) +- [ ] Total >= 60 unique keywords? (count!) +``` + +**Example 2 (stock analysis)**: +```yaml +description: This skill should be used for technical stock analysis using indicators like RSI, MACD, Bollinger Bands, moving averages. Activates when user asks about technical analysis, indicators, buy/sell signals for stocks. Supports multiple tickers, benchmark comparisons, alert generation. DO NOT use for fundamental analysis, financial statements, or news. +``` + +### Step 6: List Complete Keywords + +In SKILL.md, include complete keywords section: + +```markdown +## Keywords for Automatic Detection + +This skill is activated when user mentions: + +**Entities**: +- [complete list of organizations] +- [complete list of main objects] + +**Geography**: +- [list of countries/regions/states] + +**Metrics**: +- [list of metrics] + +**Actions**: +- [list of verbs] + +**Temporality**: +- [list of temporal terms] + +**Activation examples**: +āœ… "[example 1]" +āœ… "[example 2]" +āœ… "[example 3]" +āœ… "[example 4]" +āœ… "[example 5]" + +**Does NOT activate for**: +āŒ "[out of scope example]" +āŒ "[out of scope example]" +āŒ "[out of scope example]" +``` + +### Step 7: Mental Testing + +**Simulate detection**: + +For each example question from use cases (Phase 2), verify: +- Description contains relevant keywords? āœ… +- Doesn't contain negative scope keywords? āœ… +- Claude would detect automatically? āœ… + +**If any use case would NOT be detected**: +ā†’ Add missing keywords to description + +## Detection Design Examples + +### Example 1: US Agriculture (NASS) + +**Identified keywords**: +- Entities: USDA (5x), NASS (8x), agriculture (3x) +- Commodities: corn (12x), soybeans (10x), wheat (8x) +- Metrics: production (15x), area (10x), yield (8x) +- Geography: US (10x), states (5x), Iowa (2x) +- Actions: compare (5x), ranking (3x), trend (2x) + +**Description**: +"This skill should be used for analyses about United States agriculture using official USDA NASS data. Activates when user asks about production, area, yield of commodities like corn, soybeans, wheat. Supports YoY comparisons, rankings, trends. DO NOT use for futures or other countries." + +**Coverage**: 95% of typical use cases + +### Example 2: Global Climate (NOAA) + +**Keywords**: +- Entities: NOAA, weather, climate +- Metrics: temperature, precipitation, humidity +- Geography: global, countries, stations +- Temporality: historical, current, forecast + +**Description**: +"This skill should be used for climate analyses using NOAA data. Activates when user asks about temperature, precipitation, historical climate data or forecasts. Supports temporal and geographic aggregations, anomalies, long-term trends." + +## Phase 4 Checklist + +- [ ] Entities listed (organizations, objects, geography) +- [ ] Actions/verbs listed +- [ ] Question variations mapped +- [ ] Negative scope defined +- [ ] Description created (150-250 chars) +- [ ] Complete keywords documented in SKILL.md +- [ ] Activation examples (positive and negative) +- [ ] Mental detection simulation (all use cases covered) diff --git a/references/phase5-implementation.md b/references/phase5-implementation.md new file mode 100644 index 0000000..e0ec968 --- /dev/null +++ b/references/phase5-implementation.md @@ -0,0 +1,1011 @@ +# Phase 5: Complete Implementation + +## Objective + +**IMPLEMENT** everything with FUNCTIONAL code, USEFUL documentation, and REAL configs. + +## āš ļø FUNDAMENTAL RULES + +### What to NEVER Do + +āŒ **FORBIDDEN**: +```python +# TODO: implement this function +def analyze(): + pass +``` + +āŒ **FORBIDDEN**: +```markdown +For more details, consult the official documentation at [external link]. +``` + +āŒ **FORBIDDEN**: +```json +{ + "api_key": "YOUR_API_KEY_HERE" +} +``` + +### What to ALWAYS Do + +āœ… **MANDATORY**: Complete and functional code +āœ… **MANDATORY**: Detailed docstrings +āœ… **MANDATORY**: Useful content in references +āœ… **MANDATORY**: Configs with real values + instructions + +## Implementation Order (Updated v2.0) + +``` +1. Create directory structure +2. Create marketplace.json (MANDATORY!) +3. Implement base utils (cache, rate limiter) +4. Implement utils/helpers.py (NEW! - Temporal context) +5. Create utils/validators/ (NEW! - Validation system) +6. Implement fetch (API client - ALL endpoints!) +7. Implement parsers (1 per data type - NEW!) +8. Implement analyze (analyses + comprehensive report) +9. Create tests/ (NEW! - Test suite) +10. Create examples/ (NEW! - Real-world examples) +11. Write SKILL.md +12. Write references +13. Create assets +14. Write README +15. Create QUICK-START.md (NEW!) +16. Create CHANGELOG.md and VERSION (NEW!) +17. Create DECISIONS.md +``` + +**Why this order** (updated)? +- Marketplace.json FIRST (without it, skill can't install) +- Helpers early (used by analyze functions) +- Validators before analyze (integration) +- ALL fetch methods (not just 1!) +- Modular parsers (1 per type) +- Tests and examples before docs (validate before documenting) +- Distribution docs (QUICK-START) to facilitate usage + +## Implementation: Directory Structure + +```bash +# Create using Bash tool +mkdir -p {agent-name}/{scripts/utils,references,assets,data/{raw,processed,cache,analysis},.claude-plugin} + +# Verify +ls -la {agent-name}/ +``` + +## Implementation: Marketplace.json (MANDATORY!) + +### āš ļø CRITICAL: Without marketplace.json, skill CANNOT be installed! + +**Create FIRST**, before any other file! + +### Location + +``` +{agent-name}/.claude-plugin/marketplace.json +``` + +### Complete Template + +```json +{ + "name": "{agent-name}", + "owner": { + "name": "Agent Creator", + "email": "noreply@example.com" + }, + "metadata": { + "description": "Brief agent description", + "version": "1.0.0", + "created": "2025-10-17" + }, + "plugins": [ + { + "name": "{agent-name}-plugin", + "description": "COPY EXACTLY the description from SKILL.md frontmatter", + "source": "./", + "strict": false, + "skills": ["./"] + } + ] +} +``` + +### Required Fields + +**`name` (root level)**: +- Agent name (same as directory name) +- Example: `"climate-analysis-sorriso-mt"` + +**`plugins[0].name`**: +- Plugin name (can be agent-name + "-plugin") +- Example: `"climate-analysis-plugin"` + +**`plugins[0].description`** (VERY IMPORTANT!): +- **MUST BE EXACTLY EQUAL** to `description` in SKILL.md frontmatter +- This is the description Claude uses to detect when to activate the skill +- Copy word-for-word, including keywords +- Size: 150-250 words + +**`plugins[0].source`**: +- Always `"./"` (points to agent root) + +**`plugins[0].skills`**: +- Always `["./"]` (points to SKILL.md in root) + +### Complete Example (Climate Agent) + +```json +{ + "name": "climate-analysis-sorriso-mt", + "owner": { + "name": "Agent Creator", + "email": "noreply@example.com" + }, + "metadata": { + "description": "Climate analysis agent for Sorriso, Mato Grosso", + "version": "1.0.0", + "created": "2025-10-17" + }, + "plugins": [ + { + "name": "climate-analysis", + "description": "This skill should be used for climate analysis of Sorriso in Mato Grosso State, Brazil. Activates when user asks about temperature, precipitation, rainfall or climate in Sorriso-MT. Supports historical data analyses since 1940 including time series, year-over-year comparisons (YoY), long-term trends, climate anomaly detection, seasonal patterns and descriptive statistics. Uses Open-Meteo Historical Weather API data based on ERA5 reanalysis.", + "source": "./", + "strict": false, + "skills": ["./"] + } + ] +} +``` + +### Validation + +**After creating, ALWAYS validate**: + +```bash +# Syntax check +python -c "import json; print(json.load(open('.claude-plugin/marketplace.json')))" + +# If no error, it's valid! +``` + +**Verify**: +- āœ… JSON syntactically correct +- āœ… `plugins[0].description` **identical** to SKILL.md frontmatter +- āœ… `skills` points to `["./"]` +- āœ… `source` is `"./"` + +### Creation Order + +```bash +1. āœ… mkdir .claude-plugin +2. āœ… Write: .claude-plugin/marketplace.json ā† FIRST! +3. āœ… Write: SKILL.md (with frontmatter) +4. ... (rest of files) +``` + +### Why Marketplace.json is Mandatory + +Without this file: +- āŒ `/plugin marketplace add ./agent-name` FAILS +- āŒ Skill cannot be installed +- āŒ Claude cannot use the skill +- āŒ All work creating agent is useless + +**NEVER forget to create marketplace.json!** + +## Implementation: Python Scripts + +### Quality Standard for EVERY Script + +**Mandatory template**: + +```python +#!/usr/bin/env python3 +""" +[Script title in 1 line] + +[Detailed description in 2-3 paragraphs explaining: +- What the script does +- How it works +- When to use +- Inputs and outputs +- Dependencies] + +Example: + $ python script.py --param1 value1 --param2 value2 +""" + +# Organized imports +# 1. Standard library +import sys +import os +from pathlib import Path +from typing import Dict, List, Optional +from datetime import datetime + +# 2. Third-party +import requests +import pandas as pd + +# 3. Local +from utils.cache_manager import CacheManager + + +# Constants at top +API_BASE_URL = "https://..." +DEFAULT_TIMEOUT = 30 + + +class MainClass: + """ + [Class description] + + Attributes: + attr1: [description] + attr2: [description] + + Example: + >>> obj = MainClass(param) + >>> result = obj.method() + """ + + def __init__(self, param1: str, param2: int = 10): + """ + Initialize [MainClass] + + Args: + param1: [detailed description] + param2: [detailed description]. Defaults to 10. + + Raises: + ValueError: If param1 is invalid + """ + if not param1: + raise ValueError("param1 cannot be empty") + + self.param1 = param1 + self.param2 = param2 + + def main_method(self, input_val: str) -> Dict: + """ + [What the method does] + + Args: + input_val: [description] + + Returns: + Dict with keys: + - key1: [description] + - key2: [description] + + Raises: + APIError: If API request fails + ValueError: If input is invalid + + Example: + >>> obj.main_method("value") + {'key1': 123, 'key2': 'abc'} + """ + # Validate input + if not self._validate_input(input_val): + raise ValueError(f"Invalid input: {input_val}") + + try: + # Complete implementation here + result = self._do_work(input_val) + + return result + + except Exception as e: + # Specific error handling + print(f"Error: {e}") + raise + + def _validate_input(self, value: str) -> bool: + """Helper for validation""" + # Complete implementation + return len(value) > 0 + + def _do_work(self, value: str) -> Dict: + """Internal helper""" + # Complete implementation + return {"result": "value"} + + +def main(): + """Main function with argparse""" + import argparse + + parser = argparse.ArgumentParser( + description="[Script description]" + ) + parser.add_argument( + '--param1', + required=True, + help="[Parameter description with example]" + ) + parser.add_argument( + '--output', + default='output.json', + help="Output file path" + ) + + args = parser.parse_args() + + # Execute + obj = MainClass(args.param1) + result = obj.main_method(args.param1) + + # Save output + output_path = Path(args.output) + output_path.parent.mkdir(parents=True, exist_ok=True) + + with open(output_path, 'w') as f: + json.dump(result, f, indent=2) + + print(f"āœ“ Saved: {output_path}") + + +if __name__ == "__main__": + main() +``` + +**Verify each script has**: +- āœ… Correct shebang +- āœ… Complete module docstring +- āœ… Organized imports +- āœ… Type hints in functions +- āœ… Docstrings in classes and methods +- āœ… Error handling +- āœ… Input validations +- āœ… Main function with argparse +- āœ… if __name__ == "__main__" + +### Script 1: fetch_*.py + +**Responsibility**: API requests + +**Must implement**: + +```python +class APIClient: + """Client for [API]""" + + def __init__(self, api_key: str): + """ + Initialize API client + + Args: + api_key: API key (get from [where]) + + Raises: + ValueError: If API key is None or empty + """ + if not api_key: + raise ValueError("API key required") + + self.api_key = api_key + self.session = requests.Session() + self.rate_limiter = RateLimiter(...) + + def fetch(self, **params) -> Dict: + """ + Fetch data from API + + Args: + **params: Query parameters + + Returns: + API response as dict + + Raises: + RateLimitError: If rate limit exceeded + APIError: If API returns error + """ + # 1. Check rate limit + if not self.rate_limiter.allow(): + raise RateLimitError("Rate limit exceeded") + + # 2. Build request + url = f"{BASE_URL}/endpoint" + headers = {"Authorization": f"Bearer {self.api_key}"} + + # 3. Make request with retry + response = self._request_with_retry(url, headers, params) + + # 4. Validate response + self._validate_response(response) + + # 5. Record request + self.rate_limiter.record() + + return response.json() + + def _request_with_retry(self, url, headers, params, max_retries=3): + """Request with exponential retry""" + for attempt in range(max_retries): + try: + response = self.session.get( + url, + headers=headers, + params=params, + timeout=30 + ) + response.raise_for_status() + return response + + except requests.Timeout: + if attempt < max_retries - 1: + wait = 2 ** attempt + time.sleep(wait) + else: + raise + + except requests.HTTPError as e: + if e.response.status_code == 429: # Rate limit + raise RateLimitError("Rate limit exceeded") + elif e.response.status_code in [500, 502, 503]: + if attempt < max_retries - 1: + wait = 2 ** attempt + time.sleep(wait) + else: + raise APIError(f"API error: {e}") + else: + raise APIError(f"HTTP {e.response.status_code}: {e}") +``` + +**Size**: 200-300 lines + +### Script 2: parse_*.py + +**Responsibility**: Parsing and validation + +**Must implement**: + +```python +class DataParser: + """Parser for [API] data""" + + def parse(self, raw_data: Dict) -> pd.DataFrame: + """ + Parse raw API response to structured DataFrame + + Args: + raw_data: Raw response from API + + Returns: + Cleaned DataFrame with columns: [list] + + Raises: + ParseError: If data format is unexpected + """ + # 1. Extract data + records = raw_data.get('data', []) + + if not records: + raise ParseError("No data in response") + + # 2. Convert to DataFrame + df = pd.DataFrame(records) + + # 3. Cleaning + df = self._clean_data(df) + + # 4. Transformations + df = self._transform(df) + + # 5. Validation + self._validate(df) + + return df + + def _clean_data(self, df: pd.DataFrame) -> pd.DataFrame: + """Data cleaning""" + # Remove number formatting + for col in ['production', 'area', 'yield']: + if col in df.columns: + df[col] = df[col].str.replace(',', '') + df[col] = pd.to_numeric(df[col], errors='coerce') + + # Handle suppressed values + df = df.replace('(D)', pd.NA) + df = df.replace('(E)', pd.NA) # Or mark as estimate + + return df + + def _transform(self, df: pd.DataFrame) -> pd.DataFrame: + """Transformations""" + # Standardize names + df['state'] = df['state_name'].str.upper() + + # Convert units (if needed) + if 'unit' in df.columns and df['unit'].iloc[0] == 'BU': + # Bushels to metric tons + df['value_mt'] = df['value'] * 0.0254 + + return df + + def _validate(self, df: pd.DataFrame): + """Validations""" + # Required fields + required = ['state', 'year', 'commodity', 'value'] + missing = set(required) - set(df.columns) + if missing: + raise ParseError(f"Missing columns: {missing}") + + # Values in expected ranges + if (df['value'] < 0).any(): + raise ParseError("Negative values found") + + # No duplicates + duplicates = df.duplicated(subset=['state', 'year', 'commodity']) + if duplicates.any(): + raise ParseError(f"Duplicates found: {duplicates.sum()}") +``` + +**Size**: 150-200 lines + +### Script 3: analyze_*.py + +**Responsibility**: All analyses + +**Must implement**: + +```python +class Analyzer: + """Analyses for [domain] data""" + + def __init__(self, df: pd.DataFrame): + """ + Initialize analyzer + + Args: + df: Cleaned DataFrame from parser + """ + self.df = df + self._validate_dataframe() + + def yoy_comparison( + self, + commodity: str, + current_year: int, + previous_year: int, + geography: str = "US" + ) -> Dict: + """ + Year-over-year comparison + + Args: + commodity: Commodity name + current_year: Current year + previous_year: Previous year + geography: Geography level (US, STATE, etc) + + Returns: + Dict with comparison results: + - production_current + - production_previous + - change_absolute + - change_percent + - decomposition (if production) + - interpretation + + Example: + >>> analyzer.yoy_comparison("CORN", 2023, 2022) + {'production_current': 15.3, 'change_percent': 11.7, ...} + """ + # Filter data + df_current = self.df[ + (self.df['commodity'] == commodity) & + (self.df['year'] == current_year) & + (self.df['geography'] == geography) + ] + + df_previous = self.df[ + (self.df['commodity'] == commodity) & + (self.df['year'] == previous_year) & + (self.df['geography'] == geography) + ] + + if len(df_current) == 0 or len(df_previous) == 0: + raise ValueError(f"Data not found for {commodity} in {current_year} or {previous_year}") + + # Extract values + prod_curr = df_current['production'].iloc[0] + prod_prev = df_previous['production'].iloc[0] + + # Calculate changes + change_abs = prod_curr - prod_prev + change_pct = (change_abs / prod_prev) * 100 + + # Decomposition (if has area and yield) + decomp = None + if 'area' in df_current.columns and 'yield' in df_current.columns: + decomp = self._decompose_growth(df_current, df_previous) + + # Interpretation + if abs(change_pct) < 2: + interpretation = "stable" + elif change_pct > 10: + interpretation = "significant_increase" + elif change_pct > 2: + interpretation = "moderate_increase" + elif change_pct < -10: + interpretation = "significant_decrease" + else: + interpretation = "moderate_decrease" + + return { + "commodity": commodity, + "geography": geography, + "year_current": current_year, + "year_previous": previous_year, + "production_current": round(prod_curr, 1), + "production_previous": round(prod_prev, 1), + "change_absolute": round(change_abs, 1), + "change_percent": round(change_pct, 1), + "decomposition": decomp, + "interpretation": interpretation + } + + def _decompose_growth(self, df_current, df_previous) -> Dict: + """Area vs yield decomposition""" + area_curr = df_current['area'].iloc[0] + area_prev = df_previous['area'].iloc[0] + yield_curr = df_current['yield'].iloc[0] + yield_prev = df_previous['yield'].iloc[0] + + area_change_pct = ((area_curr - area_prev) / area_prev) * 100 + yield_change_pct = ((yield_curr - yield_prev) / yield_prev) * 100 + + prod_change_pct = ((df_current['production'].iloc[0] - df_previous['production'].iloc[0]) / + df_previous['production'].iloc[0]) * 100 + + if prod_change_pct != 0: + area_contrib = (area_change_pct / prod_change_pct) * 100 + yield_contrib = (yield_change_pct / prod_change_pct) * 100 + else: + area_contrib = yield_contrib = 0 + + return { + "area_change_pct": round(area_change_pct, 1), + "yield_change_pct": round(yield_change_pct, 1), + "area_contribution": round(area_contrib, 1), + "yield_contribution": round(yield_contrib, 1), + "growth_type": "intensive" if yield_contrib > 60 else + "extensive" if area_contrib > 60 else + "balanced" + } + + # Implement ALL other analyses + # state_ranking(), trend_analysis(), etc. + # [Complete code for each one] +``` + +**Size**: 400-500 lines + +## Implementation: SKILL.md + +### Mandatory Structure + +```markdown +--- +name: [agent-name] +description: [description of 150-250 words with all keywords] +--- + +# [Agent Name] + +[Introduction of 2-3 paragraphs explaining what the agent is] + +## When to Use This Skill + +Claude should automatically activate when user: + +āœ… [Trigger 1 with examples] +āœ… [Trigger 2 with examples] +āœ… [Trigger 3 with examples] + +## Data Source + +**API**: [Name] +**URL**: [URL] +**Documentation**: [link] +**Authentication**: [how to get key] + +[API summary in 1-2 paragraphs] + +See `references/api-guide.md` for complete details. + +## Workflows + +### Workflow 1: [Name] + +**When to execute**: [trigger conditions] + +**Step-by-step**: + +1. **Identify parameters** + - [what to extract from user's question] + +2. **Fetch data** + ```bash + python scripts/fetch_[source].py \ + --param1 value1 \ + --output data/raw/file.json + ``` + +3. **Parse data** + ```bash + python scripts/parse_[source].py \ + --input data/raw/file.json \ + --output data/processed/file.csv + ``` + +4. **Analyze** + ```bash + python scripts/analyze_[source].py \ + --input data/processed/file.csv \ + --analysis yoy \ + --output data/analysis/result.json + ``` + +5. **Interpret results** + [How to interpret the result JSON] + +**Complete example**: + +Question: "[example]" + +[Step-by-step flow with commands and outputs] + +Answer: "[expected response]" + +### Workflow 2: [Name] +[...] + +[Repeat for all main workflows] + +## Available Scripts + +### scripts/fetch_[source].py + +**Function**: Make API requests + +**Inputs**: +- `--param1`: [description] +- `--param2`: [description] + +**Output**: JSON in `data/raw/` + +**Example**: +```bash +python scripts/fetch_[source].py --commodity CORN --year 2023 +``` + +**Error handling**: +- API unavailable: [action] +- Rate limit: [action] +- [other errors] + +### scripts/parse_[source].py + +[Same level of detail...] + +### scripts/analyze_[source].py + +**Available analyses**: +- `--analysis yoy`: Year-over-year comparison +- `--analysis ranking`: State ranking +- [complete list] + +[Detail each one...] + +## Available Analyses + +### 1. YoY Comparison + +**Objective**: [...] +**When to use**: [...] +**Methodology**: [...] +**Output**: [...] +**Interpretation**: [...] + +See `references/analysis-methods.md` for detailed formulas. + +### 2. State Ranking + +[...] + +[For all analyses] + +## Error Handling + +### Error: API Unavailable + +**Symptom**: [...] +**Cause**: [...] +**Automatic action**: [...] +**Fallback**: [...] +**User message**: [...] + +### Error: Rate Limit Exceeded + +[...] + +[All expected errors] + +## Mandatory Validations + +1. **API key validation**: [...] +2. **Data validation**: [...] +3. **Consistency checks**: [...] + +## Performance and Cache + +**Cache strategy**: +- Historical data: [TTL] +- Current data: [TTL] +- Justification: [...] + +**Rate limiting**: +- Limit: [number] +- Implementation: [...] + +## References + +- `references/api-guide.md`: Complete API documentation +- `references/analysis-methods.md`: Detailed methodologies +- `references/troubleshooting.md`: Troubleshooting guide + +## Keywords for Detection + +[Complete keyword list organized by category] + +## Usage Examples + +### Example 1: [Scenario] + +**Question**: "[exact question]" + +**Internal flow**: [commands executed] + +**Answer**: "[complete and formatted answer]" + +### Examples 2-5: [...] + +[Minimum 5 complete examples] +``` + +**Size**: 5000-7000 words + +## File Creation + +**Use Write tool for each file**: + +```bash +# Creation order (UPDATED with marketplace.json): +1. āœ… Write: .claude-plugin/marketplace.json ā† MANDATORY FIRST! +2. Write: SKILL.md (frontmatter with description) +3. Write: DECISIONS.md +4. Write: scripts/utils/cache_manager.py +5. Write: scripts/utils/rate_limiter.py +6. Write: scripts/utils/validators.py +7. Write: scripts/fetch_[source].py +8. Write: scripts/parse_[source].py +9. Write: scripts/analyze_[source].py +10. Write: references/api-guide.md +11. Write: references/analysis-methods.md +12. Write: references/troubleshooting.md +13. Write: assets/config.json +14. Write: assets/metadata.json (if needed) +15. Write: README.md +16. Write: requirements.txt +17. Write: .gitignore +18. Bash: chmod +x scripts/*.py + +# āš ļø CRITICAL: marketplace.json ALWAYS FIRST! +# Reason: Without it, skill cannot be installed +``` + +## Post-Implementation Validation + +### Verify Each File + +**Python scripts**: +```bash +# Syntax check +python -m py_compile scripts/fetch_*.py +python -m py_compile scripts/parse_*.py +python -m py_compile scripts/analyze_*.py + +# Import check (mental - verify imports make sense) +``` + +**JSONs**: +```bash +# Validate syntax +python -c "import json; json.load(open('assets/config.json'))" +``` + +**Markdown**: +- [ ] SKILL.md has valid frontmatter +- [ ] No broken links +- [ ] Code blocks have syntax highlighting + +### Final Checklist + +- [ ] All files created (15+ files) +- [ ] No TODO or pass +- [ ] Code has correct imports +- [ ] JSONs are valid +- [ ] References have useful content +- [ ] README has complete instructions +- [ ] DECISIONS.md documents choices + +## Final Communication to User + +After creating everything: + +``` +āœ… AGENT CREATED SUCCESSFULLY! + +šŸ“‚ Location: ./[agent-name]/ + +šŸ“Š Statistics: +- SKILL.md: [N] words +- Python code: [N] lines +- References: [N] words +- Total files: [N] + +šŸŽÆ Main Decisions: +- API: [name] ([short justification]) +- Analyses: [list] +- Structure: [type] + +šŸ’° Estimated ROI: +- Before: [X]h/[frequency] +- After: [Y]min/[frequency] +- Savings: [%] + +šŸš€ NEXT STEPS: + +1. Get API key: + [Instructions or link] + +2. Configure: + export API_KEY_VAR="your_key" + +3. Install: + /plugin marketplace add ./[agent-name] + +4. Test: + "[example 1]" + "[example 2]" + +šŸ“– See README.md for complete instructions. +šŸ“‹ See DECISIONS.md for decision justifications. +``` + +## Phase 5 Checklist + +- [ ] Directory structure created (including .claude-plugin/) +- [ ] āœ… **marketplace.json created (MANDATORY!)** +- [ ] **marketplace.json validated (syntax + description identical to SKILL.md)** +- [ ] SKILL.md created with correct frontmatter +- [ ] DECISIONS.md created +- [ ] Utils implemented (functional code) +- [ ] Main scripts implemented (functional code) +- [ ] SKILL.md written (5000+ words) +- [ ] References written (useful content) +- [ ] Assets created (valid JSONs) +- [ ] README.md written +- [ ] requirements.txt created +- [ ] Syntax validation passed +- [ ] No TODO or placeholder +- [ ] **Test installation**: `/plugin marketplace add ./agent-name` +- [ ] Final message to user formatted diff --git a/references/phase6-testing.md b/references/phase6-testing.md new file mode 100644 index 0000000..56ddac8 --- /dev/null +++ b/references/phase6-testing.md @@ -0,0 +1,783 @@ +# Phase 6: Test Suite Generation (NEW v2.0!) + +## Objective + +**GENERATE** comprehensive test suite that validates ALL functions of the created skill. + +**LEARNING:** us-crop-monitor v1.0 had ZERO tests. When expanding to v2.0, it was difficult to ensure nothing broke. v2.0 has 25 tests (100% passing) that ensure reliability. + +--- + +## Why Are Tests Critical? + +### Benefits for Developer: +- āœ… Ensures code works before distribution +- āœ… Detects bugs early (not after client installs!) +- āœ… Allows confident changes (regression testing) +- āœ… Documents expected behavior + +### Benefits for Client: +- āœ… Confidence in skill ("100% tested") +- āœ… Fewer bugs in production +- āœ… More professional (commercially viable) + +### Benefits for Agent-Creator: +- āœ… Validates that generated skill actually works +- āœ… Catch errors before considering "done" +- āœ… Automatic quality gate + +--- + +## Test Structure + +### tests/ Directory + +``` +{skill-name}/ +ā””ā”€ā”€ tests/ + ā”œā”€ā”€ test_fetch.py # Tests API client + ā”œā”€ā”€ test_parse.py # Tests parsers + ā”œā”€ā”€ test_analyze.py # Tests analyses + ā”œā”€ā”€ test_integration.py # Tests end-to-end + ā”œā”€ā”€ test_validation.py # Tests validators + ā”œā”€ā”€ test_helpers.py # Tests helpers (year detection, etc.) + ā””ā”€ā”€ README.md # How to run tests +``` + +--- + +## Template 1: test_fetch.py + +**Objective:** Validate API client works + +```python +#!/usr/bin/env python3 +""" +Test suite for {API} client. + +Tests all fetch methods with real API data. +""" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts')) + +from fetch_{api} import {ApiClient}, DataNotFoundError + + +def test_get_{metric1}(): + """Test fetching {metric1} data.""" + print("\nTesting get_{metric1}()...") + + try: + client = {ApiClient}() + + # Test with valid parameters + result = client.get_{metric1}( + {entity}='{valid_entity}', + year=2024 + ) + + # Validations + assert 'data' in result, "Missing 'data' in result" + assert 'metadata' in result, "Missing 'metadata'" + assert len(result['data']) > 0, "No data returned" + assert result['metadata']['from_cache'] in [True, False] + + print(f" āœ“ Fetched {len(result['data'])} records") + print(f" āœ“ Metadata present") + print(f" āœ“ From cache: {result['metadata']['from_cache']}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + return False + + +def test_get_{metric2}(): + """Test fetching {metric2} data.""" + # Similar structure... + pass + + +def test_error_handling(): + """Test that errors are handled correctly.""" + print("\nTesting error handling...") + + try: + client = {ApiClient}() + + # Test invalid entity (should raise) + try: + result = client.get_{metric1}({entity}='INVALID_ENTITY', year=2024) + print(" āœ— Should have raised DataNotFoundError") + return False + except DataNotFoundError: + print(" āœ“ Correctly raises DataNotFoundError for invalid entity") + + # Test invalid year (should raise) + try: + result = client.get_{metric1}({entity}='{valid}', year=2099) + print(" āœ— Should have raised ValidationError") + return False + except Exception as e: + print(f" āœ“ Correctly raises error for future year") + + return True + + except Exception as e: + print(f" āœ— Unexpected error: {e}") + return False + + +def main(): + """Run all fetch tests.""" + print("=" * 70) + print("FETCH TESTS - {API} Client") + print("=" * 70) + + results = [] + + # Test each get_* method + results.append(("get_{metric1}", test_get_{metric1}())) + results.append(("get_{metric2}", test_get_{metric2}())) + # ... add test for ALL get_* methods + + results.append(("error_handling", test_error_handling())) + + # Summary + print("\n" + "=" * 70) + print("SUMMARY") + print("=" * 70) + + passed = sum(1 for _, r in results if r) + total = len(results) + + for name, result in results: + status = "āœ“ PASS" if result else "āœ— FAIL" + print(f"{status}: {name}()") + + print(f"\nResults: {passed}/{total} tests passed") + + return passed == total + + +if __name__ == "__main__": + success = main() + sys.exit(0 if success else 1) +``` + +**Rule:** ONE test function for EACH `get_*()` method implemented! + +--- + +## Template 2: test_parse.py + +**Objective:** Validate parsers + +```python +#!/usr/bin/env python3 +""" +Test suite for data parsers. + +Tests all parse_* modules. +""" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts')) + +from parse_{type1} import parse_{type1}_response +from parse_{type2} import parse_{type2}_response + + +def test_parse_{type1}(): + """Test {type1} parser.""" + print("\nTesting parse_{type1}_response()...") + + # Sample data (real structure from API) + sample_data = [ + { + 'field1': 'value1', + 'field2': 'value2', + 'Value': '123', + # ... real API fields + } + ] + + try: + df = parse_{type1}_response(sample_data) + + # Validations + assert not df.empty, "DataFrame is empty" + assert 'Value' in df.columns or '{metric}_value' in df.columns + assert len(df) == len(sample_data) + + print(f" āœ“ Parsed {len(df)} records") + print(f" āœ“ Columns: {list(df.columns)}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + import traceback + traceback.print_exc() + return False + + +def test_parse_empty_data(): + """Test parser handles empty data gracefully.""" + print("\nTesting empty data handling...") + + try: + from parse_{type1} import ParseError + + try: + df = parse_{type1}_response([]) + print(" āœ— Should have raised ParseError") + return False + except ParseError as e: + print(f" āœ“ Correctly raises ParseError: {e}") + return True + + except Exception as e: + print(f" āœ— Unexpected error: {e}") + return False + + +def main(): + results = [] + + # Test each parser + results.append(("parse_{type1}", test_parse_{type1}())) + results.append(("parse_{type2}", test_parse_{type2}())) + # ... for ALL parsers + + results.append(("empty_data", test_parse_empty_data())) + + # Summary + passed = sum(1 for _, r in results if r) + print(f"\nResults: {passed}/{len(results)} passed") + return passed == len(results) + + +if __name__ == "__main__": + sys.exit(0 if main() else 1) +``` + +--- + +## Template 3: test_integration.py + +**Objective:** End-to-end tests (MOST IMPORTANT!) + +```python +#!/usr/bin/env python3 +""" +Integration tests for {skill-name}. + +Tests all analysis functions with REAL API data. +""" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts')) + +from analyze_{domain} import ( + {function1}, + {function2}, + {function3}, + # ... import ALL functions +) + + +def test_{function1}(): + """Test {function1} with auto-year detection.""" + print("\n1. Testing {function1}()...") + + try: + # Test WITHOUT year (auto-detection) + result = {function1}({entity}='{valid_entity}') + + # Validations + assert 'year' in result, "Missing year" + assert 'year_requested' in result, "Missing year_requested" + assert 'year_info' in result, "Missing year_info" + assert result['year'] >= 2024, "Year too old" + assert result['year_requested'] is None, "Should auto-detect" + + print(f" āœ“ Auto-year detection: {result['year']}") + print(f" āœ“ Year info: {result['year_info']}") + print(f" āœ“ Data present: {list(result.keys())}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + import traceback + traceback.print_exc() + return False + + +def test_{function1}_with_explicit_year(): + """Test {function1} with explicit year.""" + print("\n2. Testing {function1}() with explicit year...") + + try: + # Test WITH year specified + result = {function1}({entity}='{valid_entity}', year=2024) + + assert result['year'] == 2024, f"Expected 2024, got {result['year']}" + assert result['year_requested'] == 2024 + + print(f" āœ“ Uses specified year: {result['year']}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + return False + + +def test_all_functions_exist(): + """Verify all expected functions are implemented.""" + print("\nVerifying all functions exist...") + + expected_functions = [ + '{function1}', + '{function2}', + '{function3}', + # ... ALL functions + ] + + missing = [] + for func_name in expected_functions: + if func_name not in globals(): + missing.append(func_name) + + if missing: + print(f" āœ— Missing functions: {missing}") + return False + else: + print(f" āœ“ All {len(expected_functions)} functions present") + return True + + +def main(): + """Run all integration tests.""" + print("\n" + "=" * 70) + print("{SKILL NAME} - INTEGRATION TEST SUITE") + print("=" * 70) + + results = [] + + # Test each function + results.append(("{function1} auto-year", test_{function1}())) + results.append(("{function1} explicit-year", test_{function1}_with_explicit_year())) + # ... repeat for ALL functions + + results.append(("all_functions_exist", test_all_functions_exist())) + + # Summary + print("\n" + "=" * 70) + print("FINAL SUMMARY") + print("=" * 70) + + passed = sum(1 for _, r in results if r) + total = len(results) + + print(f"\nāœ“ Passed: {passed}/{total}") + print(f"āœ— Failed: {total - passed}/{total}") + + if passed == total: + print("\nšŸŽ‰ ALL TESTS PASSED! SKILL IS PRODUCTION READY!") + else: + print(f"\nāš  {total - passed} test(s) failed - FIX BEFORE RELEASE") + + print("=" * 70) + + return passed == total + + +if __name__ == "__main__": + success = main() + sys.exit(0 if success else 1) +``` + +**Rule:** Minimum 2 tests per analysis function (auto-year + explicit-year) + +--- + +## Template 4: test_helpers.py + +**Objective:** Test year detection helpers + +```python +#!/usr/bin/env python3 +""" +Test suite for utility helpers. + +Tests temporal context detection. +""" + +import sys +from pathlib import Path +from datetime import datetime + +sys.path.insert(0, str(Path(__file__).parent.parent / 'scripts')) + +from utils.helpers import ( + get_current_{domain}_year, + should_try_previous_year, + format_year_message +) + + +def test_get_current_year(): + """Test current year detection.""" + print("\nTesting get_current_{domain}_year()...") + + try: + year = get_current_{domain}_year() + current_year = datetime.now().year + + assert year == current_year, f"Expected {current_year}, got {year}" + + print(f" āœ“ Correctly returns: {year}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + return False + + +def test_should_try_previous_year(): + """Test seasonal fallback logic.""" + print("\nTesting should_try_previous_year()...") + + try: + # Test with None (current year) + result = should_try_previous_year() + print(f" āœ“ Current year fallback: {result}") + + # Test with specific year + result_past = should_try_previous_year(2023) + print(f" āœ“ Past year fallback: {result_past}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + return False + + +def test_format_year_message(): + """Test year message formatting.""" + print("\nTesting format_year_message()...") + + try: + # Test auto-detected + msg1 = format_year_message(2025, None) + assert "auto-detected" in msg1.lower() or "2025" in msg1 + print(f" āœ“ Auto-detected: {msg1}") + + # Test requested + msg2 = format_year_message(2024, 2024) + assert "2024" in msg2 + print(f" āœ“ Requested: {msg2}") + + # Test fallback + msg3 = format_year_message(2024, 2025) + assert "not" in msg3.lower() or "fallback" in msg3.lower() + print(f" āœ“ Fallback: {msg3}") + + return True + + except Exception as e: + print(f" āœ— FAILED: {e}") + return False + + +def main(): + results = [] + + results.append(("get_current_year", test_get_current_year())) + results.append(("should_try_previous_year", test_should_try_previous_year())) + results.append(("format_year_message", test_format_year_message())) + + passed = sum(1 for _, r in results if r) + print(f"\nResults: {passed}/{len(results)} passed") + + return passed == len(results) + + +if __name__ == "__main__": + sys.exit(0 if main() else 1) +``` + +--- + +## Quality Rules for Tests + +### 1. ALL tests must use REAL DATA + +āŒ **FORBIDDEN:** +```python +def test_function(): + # Mock data + mock_data = {'fake': 'data'} + result = function(mock_data) + assert result == 'expected' +``` + +āœ… **MANDATORY:** +```python +def test_function(): + # Real API call + client = ApiClient() + result = client.get_real_data(entity='REAL', year=2024) + + # Validate real response + assert len(result['data']) > 0 + assert 'metadata' in result +``` + +**Why?** +- Tests with mocks don't guarantee API is working +- Real tests detect API changes +- Client needs to know it works with REAL data + +--- + +### 2. Tests must be FAST + +**Goal:** Complete suite in < 60 seconds + +**Techniques:** +- Use cache: First test populates cache, rest use cached +- Limit requests: Don't test 100 entities, test 2-3 +- Parallel where possible + +```python +# Example: Populate cache once +@classmethod +def setUpClass(cls): + """Populate cache before all tests.""" + client = ApiClient() + client.get_data('ENTITY1', 2024) # Cache for other tests + +# Tests then use cached data (fast) +``` + +--- + +### 3. Tests must PASS 100% + +**Quality Gate:** Skill is only "done" when ALL tests pass. + +```python +if __name__ == "__main__": + success = main() + if not success: + print("\nāŒ SKILL NOT READY - FIX FAILING TESTS") + sys.exit(1) + else: + print("\nāœ… SKILL READY FOR DISTRIBUTION") + sys.exit(0) +``` + +--- + +## Test Coverage Requirements + +### Minimum Mandatory: + +**Per module:** +- `fetch_{api}.py`: 1 test per `get_*()` method + 1 error handling test +- Each `parse_{type}.py`: 1 test per main function +- `analyze_{domain}.py`: 2 tests per analysis (auto-year + explicit-year) +- `utils/helpers.py`: 3 tests (get_year, should_fallback, format_message) + +**Expected total:** 15-30 tests depending on skill size + +**Example (us-crop-monitor v2.0):** +- test_fetch.py: 6 tests (5 get_* + 1 error) +- test_parse.py: 4 tests (4 parsers) +- test_analyze.py: 11 tests (11 functions) +- test_helpers.py: 3 tests +- test_integration.py: 1 end-to-end test +- **Total:** 25 tests + +--- + +## How to Run Tests + +### Individual: +```bash +python3 tests/test_fetch.py +python3 tests/test_integration.py +``` + +### Complete suite: +```bash +# Run all +for test in tests/test_*.py; do + python3 $test || exit 1 +done + +# Or with pytest (if available) +pytest tests/ +``` + +### In CI/CD: +```yaml +# .github/workflows/test.yml +name: Test Suite +on: [push] +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + - run: pip install -r requirements.txt + - run: python3 tests/test_integration.py +``` + +--- + +## Output Example + +**When tests pass:** +``` +====================================================================== +US CROP MONITOR - INTEGRATION TEST SUITE +====================================================================== + +1. current_condition_report()... + āœ“ Year: 2025 | Week: 39 + āœ“ Good+Excellent: 66.0% + +2. week_over_week_comparison()... + āœ“ Year: 2025 | Weeks: 39 vs 38 + āœ“ Delta: -2.2 pts + +... + +====================================================================== +FINAL SUMMARY +====================================================================== + +āœ“ Passed: 25/25 tests +āœ— Failed: 0/25 tests + +šŸŽ‰ ALL TESTS PASSED! SKILL IS PRODUCTION READY! +====================================================================== +``` + +**When tests fail:** +``` +8. yield_analysis()... + āœ— FAILED: 'yield_bu_per_acre' not in result + +... + +FINAL SUMMARY: +āœ“ Passed: 24/25 +āœ— Failed: 1/25 + +āŒ SKILL NOT READY - FIX FAILING TESTS +``` + +--- + +## Integration with Agent-Creator + +### When to generate tests: + +**In Phase 5 (Implementation):** + +Updated order: +``` +... +8. Implement analyze (analyses) +9. CREATE TESTS (ā† here!) + - Generate test_fetch.py + - Generate test_parse.py + - Generate test_analyze.py + - Generate test_helpers.py + - Generate test_integration.py +10. RUN TESTS + - Run test suite + - If fails ā†’ FIX and re-run + - Only continue when 100% passing +11. Create examples/ +... +``` + +### Quality Gate: + +```python +# Agent-creator should do: +print("Running test suite...") +exit_code = subprocess.run(['python3', 'tests/test_integration.py']).returncode + +if exit_code != 0: + print("āŒ Tests failed - aborting skill generation") + print("Fix errors above and try again") + sys.exit(1) + +print("āœ… All tests passed - continuing...") +``` + +--- + +## Testing Checklist + +Before considering skill "done": + +- [ ] tests/ directory created +- [ ] test_fetch.py with 1 test per get_*() method +- [ ] test_parse.py with 1 test per parser +- [ ] test_analyze.py with 2 tests per function (auto-year + explicit) +- [ ] test_helpers.py with year detection tests +- [ ] test_integration.py with end-to-end test +- [ ] ALL tests passing (100%) +- [ ] Test suite executes in < 60 seconds +- [ ] README in tests/ explaining how to run + +--- + +## Real Example: us-crop-monitor v2.0 + +**Tests created:** +- `test_new_metrics.py` - 5 tests (fetch methods) +- `test_year_detection.py` - 2 tests (auto-detection) +- `test_all_year_detection.py` - 4 tests (all functions) +- `test_new_analyses.py` - 3 tests (new analyses) +- `tests/test_integrated_validation.py` - 11 tests (comprehensive) + +**Total:** 25 tests, 100% passing + +**Result:** +``` +āœ“ Passed: 25/25 tests +šŸŽ‰ ALL TESTS PASSED! SKILL IS PRODUCTION READY! +``` + +**Benefit:** Full confidence v2.0 works before distribution! + +--- + +## Conclusion + +**ALWAYS generate test suite!** + +Skills without tests = prototypes +Skills with tests = professional products āœ… + +**ROI:** Tests cost +2h to create, but save 10-20h of debugging later! diff --git a/references/quality-standards.md b/references/quality-standards.md new file mode 100644 index 0000000..44bf633 --- /dev/null +++ b/references/quality-standards.md @@ -0,0 +1,937 @@ +# Mandatory Quality Standards + +## Fundamental Principles + +**Production-Ready, Not Prototype** +- Code must work without modifications +- Doesn't need "now implement X" +- Can be used immediately + +**Functional, Not Placeholder** +- Complete code in all functions +- No TODO, pass, NotImplementedError +- Robust error handling + +**Useful, Not Generic** +- Specific and detailed content +- Concrete examples, not abstract +- Not just external links + +--- + +## Standards by File Type + +### Python Scripts + +#### āœ… MANDATORY + +**1. Complete structure**: +```python +#!/usr/bin/env python3 +"""Module docstring""" + +# Imports +import ... + +# Constants +CONST = value + +# Classes/Functions +class/def ... + +# Main +def main(): + ... + +if __name__ == "__main__": + main() +``` + +**2. Docstrings**: +- Module docstring: 3-5 lines +- Class docstring: Description + Example +- Method docstring: Args, Returns, Raises, Example + +**3. Type hints**: +```python +def function(param1: str, param2: int = 10) -> Dict[str, Any]: + ... +``` + +**4. Error handling**: +```python +try: + result = risky_operation() +except SpecificError as e: + # Handle specifically + log_error(e) + raise CustomError(f"Context: {e}") +``` + +**5. Validations**: +```python +def process(data: Dict) -> pd.DataFrame: + # Validate input + if not data: + raise ValueError("Data cannot be empty") + + if 'required_field' not in data: + raise ValueError("Missing required field") + + # Process + ... + + # Validate output + assert len(result) > 0, "Result cannot be empty" + assert result['value'].notna().all(), "No null values allowed" + + return result +``` + +**6. Appropriate logging**: +```python +import logging + +logger = logging.getLogger(__name__) + +def fetch_data(): + logger.info("Fetching data from API...") + # ... + logger.debug(f"Received {len(data)} records") + # ... + logger.error(f"API error: {e}") +``` + +#### āŒ FORBIDDEN + +```python +# āŒ DON'T DO THIS: + +def analyze(): + # TODO: implement analysis + pass + +def process(data): # āŒ No type hints + # āŒ No docstring + result = data # āŒ No real logic + return result # āŒ No validation + +def fetch_api(url): + response = requests.get(url) # āŒ No timeout + return response.json() # āŒ No error handling +``` + +#### āœ… DO THIS: + +```python +def analyze_yoy(df: pd.DataFrame, commodity: str, year1: int, year2: int) -> Dict: + """ + Perform year-over-year analysis + + Args: + df: DataFrame with parsed data + commodity: Commodity name (e.g., "CORN") + year1: Current year + year2: Previous year + + Returns: + Dict with keys: + - production_current: float + - production_previous: float + - change_percent: float + - interpretation: str + + Raises: + ValueError: If data not found for specified years + DataQualityError: If data fails validation + + Example: + >>> analyze_yoy(df, "CORN", 2023, 2022) + {'production_current': 15.3, 'change_percent': 11.7, ...} + """ + # Validate inputs + if commodity not in df['commodity'].unique(): + raise ValueError(f"Commodity {commodity} not found in data") + + # Filter data + df1 = df[(df['commodity'] == commodity) & (df['year'] == year1)] + df2 = df[(df['commodity'] == commodity) & (df['year'] == year2)] + + if len(df1) == 0 or len(df2) == 0: + raise ValueError(f"Data not found for {commodity} in {year1} or {year2}") + + # Extract values + prod1 = df1['production'].iloc[0] + prod2 = df2['production'].iloc[0] + + # Calculate + change = prod1 - prod2 + change_pct = (change / prod2) * 100 + + # Interpret + if abs(change_pct) < 2: + interpretation = "stable" + elif change_pct > 10: + interpretation = "significant_increase" + elif change_pct > 2: + interpretation = "moderate_increase" + elif change_pct < -10: + interpretation = "significant_decrease" + else: + interpretation = "moderate_decrease" + + # Return + return { + "commodity": commodity, + "production_current": round(prod1, 1), + "production_previous": round(prod2, 1), + "change_absolute": round(change, 1), + "change_percent": round(change_pct, 1), + "interpretation": interpretation + } +``` + +--- + +### SKILL.md + +#### āœ… MANDATORY + +**1. Valid frontmatter**: +```yaml +--- +name: agent-name +description: [150-250 words with keywords] +--- +``` + +**2. Size**: 5000-7000 words + +**3. Mandatory sections**: +- When to use (specific triggers) +- Data source (detailed API) +- Workflows (complete step-by-step) +- Scripts (each one explained) +- Analyses (methodologies) +- Errors (complete handling) +- Validations (mandatory) +- Keywords (complete list) +- Examples (5+ complete) + +**4. Detailed workflows**: + +āœ… **GOOD**: +```markdown +### Workflow: YoY Comparison + +1. **Identify question parameters** + - Commodity: [extract from question] + - Years: Current vs previous (or specified) + +2. **Fetch data** + ```bash + python scripts/fetch_nass.py \ + --commodity CORN \ + --years 2023,2022 \ + --output data/raw/corn_2023_2022.json + ``` + +3. **Parse** + ```bash + python scripts/parse_nass.py \ + --input data/raw/corn_2023_2022.json \ + --output data/processed/corn.csv + ``` + +4. **Analyze** + ```bash + python scripts/analyze_nass.py \ + --input data/processed/corn.csv \ + --analysis yoy \ + --commodity CORN \ + --year1 2023 \ + --year2 2022 \ + --output data/analysis/corn_yoy.json + ``` + +5. **Interpret results** + + File `data/analysis/corn_yoy.json` contains: + ```json + { + "production_current": 15.3, + "change_percent": 11.7, + "interpretation": "significant_increase" + } + ``` + + Respond to user: + "Corn production grew 11.7% in 2023..." +``` + +āŒ **BAD**: +```markdown +### Workflow: Comparison + +1. Get data +2. Compare +3. Return result +``` + +**5. Complete examples**: + +āœ… **GOOD**: +```markdown +### Example 1: YoY Comparison + +**Question**: "How's corn production compared to last year?" + +**Executed flow**: +[Specific commands with outputs] + +**Generated answer**: +"Corn production in 2023 is 15.3 billion bushels, +growth of 11.7% vs 2022 (13.7 billion). Growth +comes mainly from area increase (+8%) with stable yield." +``` + +āŒ **BAD**: +```markdown +### Example: Comparison + +User asks about comparison. Agent compares and responds. +``` + +#### āŒ FORBIDDEN + +- Empty sections +- "See documentation" +- Workflows without specific commands +- Generic examples + +--- + +### References + +#### āœ… MANDATORY + +**1. Useful and self-contained content**: + +āœ… **GOOD** (references/api-guide.md): +```markdown +## Endpoint: Get Production Data + +**URL**: `GET https://quickstats.nass.usda.gov/api/api_GET/` + +**Parameters**: +- `commodity_desc`: Commodity name + - Example: "CORN", "SOYBEANS" + - Case-sensitive +- `year`: Desired year + - Example: 2023 + - Range: 1866-present + +**Complete request example**: +```bash +curl -H "X-Api-Key: YOUR_KEY" \ + "https://quickstats.nass.usda.gov/api/api_GET/?commodity_desc=CORN&year=2023&format=JSON" +``` + +**Expected response**: +```json +{ + "data": [ + { + "year": 2023, + "commodity_desc": "CORN", + "value": "15,300,000,000", + "unit_desc": "BU" + } + ] +} +``` + +**Important fields**: +- `value`: Comes as STRING with commas + - Solution: `value.replace(',', '')` + - Convert to float after +``` + +āŒ **BAD**: +```markdown +## API Endpoint + +For details on how to use the API, consult the official documentation at: +https://quickstats.nass.usda.gov/api + +[End of file] +``` + +**2. Adequate size**: +- API guide: 1500-2000 words +- Analysis methods: 2000-3000 words +- Troubleshooting: 1000-1500 words + +**3. Concrete examples**: +- Always include examples with real values +- Executable code blocks +- Expected outputs + +#### āŒ FORBIDDEN + +- "For more information, see [link]" +- Sections with only 2-3 lines +- Lists without details +- Circular references ("see other doc that sees other doc") + +--- + +### Assets (Configs) + +#### āœ… MANDATORY + +**1. Syntactically valid JSON**: +```bash +# ALWAYS validate: +python -c "import json; json.load(open('config.json'))" +``` + +**2. Real values**: + +āœ… **GOOD**: +```json +{ + "api": { + "base_url": "https://quickstats.nass.usda.gov/api", + "api_key_env": "NASS_API_KEY", + "_instructions": "Get free API key from: https://quickstats.nass.usda.gov/api#registration", + "rate_limit_per_day": 1000, + "timeout_seconds": 30 + } +} +``` + +āŒ **BAD**: +```json +{ + "api": { + "base_url": "YOUR_API_URL_HERE", + "api_key": "YOUR_KEY_HERE" + } +} +``` + +**3. Inline comments** (using `_comment` or `_note`): +```json +{ + "_comment": "Differentiated TTL by data type", + "cache": { + "ttl_historical_days": 365, + "_note_historical": "Historical data doesn't change", + "ttl_current_days": 7, + "_note_current": "Current year data may be revised" + } +} +``` + +--- + +### README.md + +#### āœ… MANDATORY + +**1. Complete installation instructions**: + +āœ… **GOOD**: +```markdown +## Installation + +### 1. Get API Key (Free) + +1. Access https://quickstats.nass.usda.gov/api#registration +2. Fill form: + - Name: [your name] + - Email: [your email] + - Purpose: "Personal research" +3. Click "Submit" +4. You'll receive email with API key in ~1 minute +5. Key format: `A1B2C3D4-E5F6-G7H8-I9J0-K1L2M3N4O5P6` + +### 2. Configure Environment + +**Option A - Export** (temporary): +```bash +export NASS_API_KEY="your_key_here" +``` + +**Option B - .bashrc/.zshrc** (permanent): +```bash +echo 'export NASS_API_KEY="your_key_here"' >> ~/.bashrc +source ~/.bashrc +``` + +**Option C - .env file** (per project): +```bash +echo "NASS_API_KEY=your_key_here" > .env +``` + +### 3. Install Dependencies + +```bash +cd nass-usda-agriculture +pip install -r requirements.txt +``` + +Requirements: +- requests +- pandas +- numpy +``` + +āŒ **BAD**: +```markdown +## Installation + +1. Get API key from the official website +2. Configure environment +3. Install dependencies +4. Done! +``` + +**2. Concrete usage examples**: + +āœ… **GOOD**: +```markdown +## Examples + +### Example 1: Current Production + +``` +You: "What's US corn production in 2023?" + +Claude: "Corn production in 2023 was 15.3 billion +bushels (389 million metric tons)..." +``` + +### Example 2: YoY Comparison + +``` +You: "Compare soybeans this year vs last year" + +Claude: "Soybean production in 2023 is 2.6% below 2022: +- 2023: 4.165 billion bushels +- 2022: 4.276 billion bushels +- Drop from area (-4.5%), yield improved (+0.8%)" +``` + +[3-5 more examples] +``` + +āŒ **BAD**: +```markdown +## Usage + +Ask questions about agriculture and the agent will respond. +``` + +**3. Specific troubleshooting**: + +āœ… **GOOD**: +```markdown +### Error: "NASS_API_KEY environment variable not found" + +**Cause**: API key not configured + +**Step-by-step solution**: +1. Verify key was obtained: https://... +2. Configure environment: + ```bash + export NASS_API_KEY="your_key_here" + ``` +3. Verify: + ```bash + echo $NASS_API_KEY + ``` +4. Should show your key +5. If doesn't work, restart terminal + +**Still not working?** +- Check for extra spaces in key +- Verify key hasn't expired (validity: 1 year) +- Re-generate key if needed +``` + +--- + +## Quality Checklist + +### Per Python Script + +- [ ] Shebang: `#!/usr/bin/env python3` +- [ ] Module docstring (3-5 lines) +- [ ] Organized imports (stdlib, 3rd party, local) +- [ ] Constants at top (if applicable) +- [ ] Type hints in all public functions +- [ ] Docstrings in classes (description + attributes + example) +- [ ] Docstrings in methods (Args, Returns, Raises, Example) +- [ ] Error handling for risky operations +- [ ] Input validations +- [ ] Output validations +- [ ] Appropriate logging +- [ ] Main function with argparse +- [ ] if __name__ == "__main__" +- [ ] Functional code (no TODO/pass) +- [ ] Valid syntax (test: `python -m py_compile script.py`) + +### Per SKILL.md + +- [ ] Frontmatter with name and description +- [ ] Description 150-250 characters with keywords +- [ ] Size 5000+ words +- [ ] "When to Use" section with specific triggers +- [ ] "Data Source" section detailed +- [ ] Step-by-step workflows with commands +- [ ] Scripts explained individually +- [ ] Analyses documented (objective, methodology) +- [ ] Errors handled (all expected) +- [ ] Validations listed +- [ ] Performance/cache explained +- [ ] Complete keywords +- [ ] Complete examples (5+) + +### Per Reference File + +- [ ] 1000+ words +- [ ] Useful content (not just links) +- [ ] Concrete examples with real values +- [ ] Executable code blocks +- [ ] Well structured (headings, lists) +- [ ] No empty sections +- [ ] No "TODO: write" + +### Per Asset (Config) + +- [ ] Syntactically valid JSON (validate!) +- [ ] Real values (not "YOUR_X_HERE" without context) +- [ ] Inline comments (_comment, _note) +- [ ] Instructions for values user must fill +- [ ] Logical and organized structure + +### Per README.md + +- [ ] Step-by-step installation +- [ ] How to get API key (detailed) +- [ ] How to configure (3 options) +- [ ] How to install dependencies +- [ ] How to install in Claude Code +- [ ] Usage examples (5+) +- [ ] Troubleshooting (10+ problems) +- [ ] License +- [ ] Contact/contribution (if applicable) + +### Complete Agent + +- [ ] DECISIONS.md documents all choices +- [ ] **VERSION** file created (e.g. 1.0.0) +- [ ] **CHANGELOG.md** created with complete v1.0.0 entry +- [ ] **INSTALACAO.md** with complete didactic tutorial +- [ ] **comprehensive_{domain}_report()** implemented +- [ ] marketplace.json with version field +- [ ] 18+ files created +- [ ] ~1500+ lines of Python code +- [ ] ~10,000+ words of documentation +- [ ] 2+ configs +- [ ] requirements.txt +- [ ] .gitignore (if needed) +- [ ] No placeholder/TODO +- [ ] Valid syntax (Python, JSON, YAML) +- [ ] Ready to use (production-ready) + +--- + +## Quality Examples + +### Example: Error Handling + +āŒ **BAD**: +```python +def fetch(url): + return requests.get(url).json() +``` + +āœ… **GOOD**: +```python +def fetch(url: str, timeout: int = 30) -> Dict: + """ + Fetch data from URL with error handling + + Args: + url: URL to fetch + timeout: Timeout in seconds + + Returns: + JSON response as dict + + Raises: + NetworkError: If connection fails + TimeoutError: If request times out + APIError: If API returns error + """ + try: + response = requests.get(url, timeout=timeout) + response.raise_for_status() + + data = response.json() + + if 'error' in data: + raise APIError(f"API error: {data['error']}") + + return data + + except requests.Timeout: + raise TimeoutError(f"Request timed out after {timeout}s") + + except requests.ConnectionError as e: + raise NetworkError(f"Connection failed: {e}") + + except requests.HTTPError as e: + if e.response.status_code == 429: + raise RateLimitError("Rate limit exceeded") + else: + raise APIError(f"HTTP {e.response.status_code}: {e}") +``` + +### Example: Validations + +āŒ **BAD**: +```python +def parse(data): + df = pd.DataFrame(data) + return df +``` + +āœ… **GOOD**: +```python +def parse(data: List[Dict]) -> pd.DataFrame: + """Parse and validate data""" + + # Validate input + if not data: + raise ValueError("Data cannot be empty") + + if not isinstance(data, list): + raise TypeError(f"Expected list, got {type(data)}") + + # Parse + df = pd.DataFrame(data) + + # Validate schema + required_cols = ['year', 'commodity', 'value'] + missing = set(required_cols) - set(df.columns) + if missing: + raise ValueError(f"Missing required columns: {missing}") + + # Validate types + df['year'] = pd.to_numeric(df['year'], errors='raise') + df['value'] = pd.to_numeric(df['value'], errors='raise') + + # Validate ranges + current_year = datetime.now().year + if (df['year'] > current_year).any(): + raise ValueError(f"Future years found (max allowed: {current_year})") + + if (df['value'] < 0).any(): + raise ValueError("Negative values found") + + # Validate no duplicates + if df.duplicated(subset=['year', 'commodity']).any(): + raise ValueError("Duplicate records found") + + return df +``` + +### Example: Docstrings + +āŒ **BAD**: +```python +def analyze(df, commodity): + """Analyze data""" + # ... +``` + +āœ… **GOOD**: +```python +def analyze_yoy( + df: pd.DataFrame, + commodity: str, + year1: int, + year2: int +) -> Dict[str, Any]: + """ + Perform year-over-year comparison analysis + + Compares production, area, and yield between two years + and decomposes growth into area vs yield contributions. + + Args: + df: DataFrame with columns ['year', 'commodity', 'production', 'area', 'yield'] + commodity: Commodity name (e.g., "CORN", "SOYBEANS") + year1: Current year to compare + year2: Previous year to compare against + + Returns: + Dict containing: + - production_current (float): Production in year1 (million units) + - production_previous (float): Production in year2 + - change_absolute (float): Absolute change + - change_percent (float): Percent change + - decomposition (dict): Area vs yield contribution + - interpretation (str): "increase", "decrease", or "stable" + + Raises: + ValueError: If commodity not found in data + ValueError: If either year not found in data + DataQualityError: If production != area * yield (tolerance > 1%) + + Example: + >>> df = pd.DataFrame([ + ... {'year': 2023, 'commodity': 'CORN', 'production': 15.3, 'area': 94.6, 'yield': 177}, + ... {'year': 2022, 'commodity': 'CORN', 'production': 13.7, 'area': 89.2, 'yield': 173} + ... ]) + >>> result = analyze_yoy(df, "CORN", 2023, 2022) + >>> result['change_percent'] + 11.7 + """ + # [Complete implementation] +``` + +--- + +## Anti-Patterns + +### Anti-Pattern 1: Partial Implementation + +āŒ **NO**: +```python +def yoy_comparison(df, commodity, year1, year2): + # Implement YoY comparison + pass + +def state_ranking(df, commodity): + # TODO: implement ranking + raise NotImplementedError() +``` + +āœ… **YES**: +```python +# [Complete and functional code for BOTH functions] +``` + +### Anti-Pattern 2: Empty References + +āŒ **NO**: +```markdown +# Analysis Methods + +## YoY Comparison + +This method compares two years. + +## Ranking + +This method ranks states. +``` + +āœ… **YES**: +```markdown +# Analysis Methods + +## YoY Comparison + +### Objective +Compare metrics between current and previous year... + +### Detailed Methodology + +**Formulas**: +``` +Ī” X = X(t) - X(t-1) +Ī” X% = (Ī” X / X(t-1)) Ɨ 100 +``` + +**Decomposition** (for production): +[Complete mathematics] + +**Interpretation**: +- |Ī”| < 2%: Stable +- Ī” > 10%: Significant increase +[...] + +### Validations +[List] + +### Complete Numerical Example +[With real values] +``` + +### Anti-Pattern 3: Useless Configs + +āŒ **NO**: +```json +{ + "api_url": "INSERT_URL", + "api_key": "INSERT_KEY" +} +``` + +āœ… **YES**: +```json +{ + "_comment": "Configuration for NASS USDA Agent", + "api": { + "base_url": "https://quickstats.nass.usda.gov/api", + "_note": "This is the official USDA NASS API base URL", + "api_key_env": "NASS_API_KEY", + "_key_instructions": "Get free API key from: https://quickstats.nass.usda.gov/api#registration" + } +} +``` + +--- + +## Final Validation + +Before delivering to user, verify: + +### Sanity Test + +```bash +# 1. Python syntax +find scripts -name "*.py" -exec python -m py_compile {} \; + +# 2. JSON syntax +python -c "import json; json.load(open('assets/config.json'))" + +# 3. Imports make sense +grep -r "^import\|^from" scripts/*.py | sort | uniq +# Verify all libs are: stdlib, requests, pandas, numpy +# No imports of uninstalled libs + +# 4. SKILL.md has frontmatter +head -5 SKILL.md | grep "^---$" + +# 5. SKILL.md size +wc -w SKILL.md +# Should be > 5000 words +``` + +### Final Checklist + +- [ ] Syntax check passed (Python, JSON) +- [ ] No import of non-existent lib +- [ ] No TODO or pass +- [ ] SKILL.md > 5000 words +- [ ] References with content +- [ ] README with complete instructions +- [ ] DECISIONS.md created +- [ ] requirements.txt created