Himalayan GPT for Restaurant Ordering Chatbots: Benchmarking Intent Classification, High Inference Latency, and Free Generation in Low-Resource Nepali

Himalayan GPT for Restaurant Ordering Chatbots: Benchmarking Intent Classification, High Inference Latency, and Free Generation in Low-Resource Nepali

Authors: Dimanjan Dahal, Kushal Shrestha, Gemini Antigravity
Project: Himalayan GPT / Restaurant Ordering Research
Date: 22 Jul 2026
Workspace: /home/venom/Desktop/food_restaurant/himalayan-gpt
Document Classification: Empirical Research Benchmark

1. Executive Summary

Automating conversational commerce in low-resource languages such as Nepali (and its commonly typed Romanized vernacular, *Roman Nepali*) requires models capable of both understanding regional culinary nomenclature and responding in sub-second conversational windows.

In this investigation, two specialized evaluation scripts were developed and benchmarked:

1. eval_restaurant_intent.py: A structured intent-classification benchmark evaluating the instruction-tuned model himalaya-ai/himalaya-gemma-4-e2b-it across 13 canonical restaurant ordering utterances.


2. probe_nepali_generation.py: A minimal free-generation smoke test probing open-domain question answering and generation stability in the lightweight foundation model himalaya-ai/himalayagpt-0.5b.

The second experiment revealed severe decoding degradation: when queried with the elementary factual prompt *"nepal ko rajdhani?"* (What is the capital of Nepal?), himalayagpt-0.5b failed completely and degenerated into an infinite loop of repetitive token sequences (*"ko ko ko..."*).

Together, these findings validate the instruction-following promise of Himalayan GPT architectures for low-resource intent classification, while simultaneously establishing that inference latency, hardware offloading bottlenecks, and generation reliability must be systematically solved before production deployment in restaurant ordering agents.

2. Methodology

2.1 Structured Intent Classification Pipeline (eval_restaurant_intent.py)

• Model Load & Initialization: Loads himalaya-ai/himalaya-gemma-4-e2b-it using PyTorch bfloat16 precision.
• System Prompt Framing: Injects an explicit restaurant-domain system prompt specifying 11 predefined intent labels (place_order, add_item, remove_item, modify_item, ask_price, ask_menu, repeat_order, cancel_order, confirm_order, greeting, unknown).
• Deterministic Decoding: Enforces do_sample=False (greedy argmax decoding) to eliminate stochastic variation across runs.
• JSON Schema Constraint: Mandates output parseable as {"intent": "", "confidence": float, "reason": ""}.
• Instrumentation: Runs a warmup request, profiles 13 labeled Roman Nepali test utterances, logs per-query wall-clock latency, records CPU/RAM/VRAM memory telemetry before and after evaluation, and opens an interactive console loop for exploratory probing.

2.2 Open-Ended Free Generation Probe (probe_nepali_generation.py)

• Loads the smaller causal language model himalaya-ai/himalayagpt-0.5b across 113 weight shards.


• Dispatches an unconstrained prompt (*"nepal ko rajdhani?"*) without instruction prefixes, sampling penalties, or task formatting, measuring raw baseline generative coherence in native Nepali contexts.

3. Runtime & Hardware Environment

All evaluations were executed on a dedicated local Linux workstation equipped with an 8 GB-class laptop GPU:

4. Quantitative Findings: eval_restaurant_intent.py

4.1 Benchmark Classification Accuracy

4.2 Latency Distribution Summary

• Total Test Set Accuracy: 0.9231 (12 correct out of 13)
• Mean Per-Query Latency: 39.6046 s
• Minimum Latency: 32.7020 s
• Maximum Latency: 46.6634 s

5. Intelligent Analysis: The 39.6-Second Latency Bottleneck

5.1 Telemetry During Model Load vs Post-Evaluation

5.2 Root Cause: CPU/Disk Parameter Offloading

Some parameters were placed on the meta device and offloaded to disk and CPU.

Although the laptop GPU reported 5.26 GB of free VRAM, the initialization scripts triggered automatic device mapping with disk and CPU offload tiers.

Whenever an autoregressive model offloads layer weights to host system memory or NVMe storage:

1. PCIe Bus Thrashing: For every decoded token, offloaded weight tensors must be transferred across the PCIe bus from RAM to VRAM.


2. Autoregressive Multiplication: In a 30-token JSON generation, transferring gigabytes of weights 30 times consecutively results in cumulative transfer overheads that dwarf actual matrix multiplication.


3. Latency Explosion: This accounts directly for the 39.60-second mean latency—a figure entirely incompatible with real-time Messenger, WhatsApp, or voice-based restaurant ordering bots.

+-------------------------------------------------------------------------+
|                  THE OFF-LOAD LATENCY AMPLIFICATION LOOP                 |
|                                                                         |
|  Token t=1:  [Host RAM/Disk] --(PCIe Transfer)--> [VRAM] -> Forward -> Logit  |
|  Token t=2:  [Host RAM/Disk] --(PCIe Transfer)--> [VRAM] -> Forward -> Logit  |
|  ...                                                                    |
|  Token t=35: [Host RAM/Disk] --(PCIe Transfer)--> [VRAM] -> Forward -> Logit  |
|                                                                         |
|  Total Cumulative Delay = 35 sequential memory transfers = ~39.6 seconds  |
+-------------------------------------------------------------------------+

6. Interactive Exploratory Observations

During the interactive session of eval_restaurant_intent.py, free-form utterances were supplied to evaluate contextual flexibility:

7. Findings: probe_nepali_generation.py

The lightweight smoke-test script evaluated himalaya-ai/himalayagpt-0.5b:

Prompt: "nepal ko rajdhani?"
Model Output: "nepal ko rajdhani? ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko koko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko"

Architectural Diagnostics:

• Model Size: himalayagpt-0.5b loaded across 113 shards.
• Execution Latency: ~14 seconds total runtime.
• Deprecation Warning: Logged torch_dtype is deprecated; dtype should be used instead.
• Degeneration Mechanism: In the absence of repetition penalties, temperature dampening, or instruction-following chat formatting, the sub-billion model experienced complete probability collapse on the high-frequency Nepali particle *"ko"*.

8. Comparative Analysis: Maturity Matrix

9. Limitations of the Current Research

1. Test Set Scope: 13 hand-crafted utterances cannot statistically represent the diversity, dialectal noise, and typographical variances across Nepal.
2. Artificial Confidence Score: The model predicted a static confidence: 0.95 across all 13 queries—including the wrong weather prediction. The model is completely uncalibrated for uncertainty.
3. Absence of Conversation State: Single-turn tests do not assess whether the model can maintain an active cart across multi-turn conversational updates.
4. Decoding Strategy: Greedy decoding prevented token loops in the Gemma model, but neither beam search, contrastive decoding, nor speculative sampling were benchmarked.

10. Deployment Risks & Engineering Recommendations

10.1 Key Risks

• User Frustration from Latency: 40-second response latency will cause over 90% user drop-off in Messenger/WhatsApp channels.
• False Rejection in Food Customization: Misclassifying flavor modifications as new orders risks placing duplicate orders in kitchen point-of-sale (POS) systems.
• Off-Topic Vulnerability: Unrelated queries being accepted as restaurant actions can trigger erroneous payment or booking flows.

10.2 Recommended Engineering Mitigations

1. Eliminate Parameter Offloading via Quantization:

1. Adopt Constrained Classification Heads Instead of Generative LLMs:

1. Expand Evaluation Harness:

1. Implement Fallback & Calibration Thresholds:

11. Conclusion

This benchmark provides clear empirical evidence: Himalayan GPT demonstrates strong linguistic intent recognition (92.31%) in Roman Nepali, but current unquantized offloaded deployments suffer from prohibitive latency (~39.6s) and unconstrained foundation models exhibit severe generational instability.

By transitioning from autoregressive generation to 4-bit resident quantization or specialized classification heads, Nepali conversational ordering assistants can achieve both high semantic accuracy and real-time execution speeds.

Appendix: Key Raw Experimental Outputs

Failure Case 1: Out-of-Distribution Miss (eval_restaurant_intent.py)

[MISS] 'today weather kasto cha' -> greeting (expected=unknown, conf=0.95, latency=38.3682s)

Failure Case 2: Degenerative Text Collapse (probe_nepali_generation.py)

Prompt: "nepal ko rajdhani?"
Output: "nepal ko rajdhani? ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko koko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko ko"