Research Engineering
HITL Multi-Evidence ESL Essay Assessment System
A local desktop research system for ESL essay assessment that combines linguistic feature extraction, local LLM rubric evaluation, and human review.
Python · Electron · Ollama · spaCy · NLTK · LanguageTool · Local LLM Inference
Status
Implemented and ready for experimental use. The associated research, "Beyond Prompt Engineering: A Human-in-the-Loop (HITL) Multi-Evidence Framework for ESL Essay Assessment," has been accepted for presentation at the 23rd AsiaCALL International Conference in Hanoi, Vietnam, November 14-15, 2026.
Overview
I am primarily responsible for building the technical system that operationalizes the research framework. The application combines essay ingestion, linguistic feature extraction, local LLM rubric evaluation, progress tracking, human review, final score entry, decision notes, and workbook export.
Problem
The research needed more than prompt experiments: it needed a working system that could process essays, preserve evidence, support human decision notes, and export completed workbook results.
Requirements
- Run locally/offline instead of relying on cloud model inference
- Process essay workbooks and preserve scoring progress
- Extract linguistic evidence using NLP tools
- Run local LLM rubric evaluation through Ollama
- Let human reviewers compare evidence, assign final scores, and record decision notes
- Export completed workbook results for experimental use
Architecture
Technical Decisions
Local-first architecture
The system uses a desktop frontend and local backend services so evaluation can happen without sending essays to cloud inference APIs.
Multi-evidence scoring
The workflow keeps linguistic features, model judgments, and human review visible as separate evidence sources instead of collapsing everything into one opaque score.
Human final decision
The system supports local LLM evaluation, but final scoring and decision notes stay in the human reviewer workflow.
Implementation
The system coordinates an Electron desktop interface with a local HTTP API and Python backend. The backend integrates spaCy, NLTK, LanguageTool, Ollama, and local Llama inference to produce evidence for human review. Reviewers can inspect essays, compare linguistic evidence with model judgments, enter final scores, document decision notes, and export a completed workbook.
Challenges, Tradeoffs, and Results
The main tradeoff is system complexity: local inference and multi-tool NLP integration require more orchestration than a cloud-only prompt prototype. The benefit is a research workflow that preserves evidence, supports human judgment, and can operate locally/offline. The human-vs-HITL experiment has not yet been completed, so the site does not claim accuracy, consistency, or efficiency improvements.