This record accompanies the paper \"Benchmarking Local Language Models for Social Robots using Edge Devices\" [accepted IEEE ARSO 2026] and contains the raw benchmark data, MMLU scores, automated teaching-effectiveness ratings, human rater sheets, and the analysis notebook supporting the results reported therein.
\nWe benchmarked 25 open-source language models for local deployment on edge hardware in a social-educational robotics context (the Robot Study Companion project, rsc.ee). Each model was evaluated across three dimensions — inference efficiency (tokens per second, energy consumption), general knowledge (a six-category MMLU subset), and teaching effectiveness (LLM-rated pedagogical quality validated against five independent human raters) — primarily on the Raspberry Pi 4, with scalability comparisons on the Raspberry Pi 5 and a laptop NVIDIA RTX 4060 GPU.
\nThis record contains: per-query hardware telemetry across all three platforms, per-model MMLU scores, GPT-4o-mini teaching-effectiveness ratings, five human rater workbooks, and the notebook that computes inter-rater agreement statistics. Readers should consult the paper for methodology, results, and discussion; this record serves as the underlying evidentiary base.
\n.\n\u251c\u2500\u2500 benchmarks/\n\u2502 \u251c\u2500\u2500 benchmarks_merged.csv 727-row consolidated per-query telemetry across all platforms\n\u2502 \u251c\u2500\u2500 results_pi4/ 7 CSV files, per-query benchmarks on Raspberry Pi 4\n\u2502 \u251c\u2500\u2500 results_pi5/ 3 CSV files, per-query benchmarks on Raspberry Pi 5\n\u2502 \u2514\u2500\u2500 results_computer/ 24 CSV files, per-query benchmarks on laptop GPU
\u2502 \u2514\u2500\u2500 fig1_final.pdf Figure 1 (benchmark summary)\n\u251c\u2500\u2500 MMLU/\n\u2502 \u251c\u2500\u2500 MMLU_merged.csv 25-row merged MMLU results (model tags harmonised with benchmarks)\n\u2502 \u2514\u2500\u2500 models_MMLU_scores/ 25 paired CSV+JSON files, per-model MMLU results\n\u2514\u2500\u2500 rated_teaching/\n \u251c\u2500\u2500 human_ratings_merged.csv 200-row merged human rater workbook data, deblinded\n \u251c\u2500\u2500 teaching_effectiveness_ratings/ GPT-4o-mini per-response ratings (250 rows)\n \u2514\u2500\u2500 human_rate_gpt4o/ 5 rater workbooks + analysis\n \u251c\u2500\u2500 annotation_analysis.ipynb human-rater analysis (α, ICC, Pearson r, Fig 2)\n \u2514\u2500\u2500 figure_2_human_annotation.pdf Figure 2 (human annotation validation)benchmarks/results_*/benchmark_all_models_*.csvPer-query records from benchmark runs. Each row captures one model answering one question, with full response text and hardware telemetry. Filenames encode the run timestamp: benchmark_all_models_YYYYMMDD_HHMMSS.csv.
Shared columns (all platforms, 18 fields):
\n| Column | \nDescription | \n
|---|---|
timestamp | \nISO-8601 start of inference | \n
model | \nOllama model tag (e.g. qwen3:0.6b) | \n
model_parameters | \nNominal parameter count, as reported | \n
question | \nBenchmark prompt text (see Table I in paper) | \n
response | \nFull model-generated answer (not truncated) | \n
response_length_chars | \nCharacter count of the response | \n
estimated_tokens | \nToken count estimated from streaming chunks (see caveat) | \n
inference_time_s | \nTotal generation time, via time.time() | \n
time_to_first_token_s | \nLatency to first streamed chunk, via time.time() | \n
tokens_per_second | \nThroughput | \n
cpu_baseline_percent, cpu_average_percent | \nCPU load (pre-inference baseline; average during inference) | \n
cpu_per_core | \nPer-core utilisation, stringified Python list; parse with ast.literal_eval | \n
cpu_freq_mhz | \nCPU frequency during inference | \n
memory_baseline_mb, memory_peak_mb, memory_increase_mb, memory_percent | \nRAM telemetry | \n
Raspberry Pi-only additional columns (19 fields, present in results_pi4/ and results_pi5/):
| Column | \nDescription | \n
|---|---|
temperature_c | \nCPU die temperature (vcgencmd measure_temp) | \n
throttled | \nThrottling flag (vcgencmd get_throttled) | \n
avg_voltage_v | \nMean rail voltage during inference (vcgencmd measure_volts) | \n
estimated_current_a | \nCurrent estimate from linear CPU-load model (idle 0.6A, full-load 3A at 5V) | \n
avg_power_watts | \nV × A, averaged over inference | \n
total_energy_joules | \nEstimated energy consumption during inference | \n
tokens_per_joule | \nEnergy efficiency metric (estimated_tokens / total_energy_joules) | \n
io_read_count, io_write_count, io_read_bytes, io_write_bytes, io_read_time_ms, io_write_time_ms, io_total_ops, io_total_bytes, io_iops, io_throughput_mb_s, io_avg_read_latency_ms, io_avg_write_latency_ms | \nDisk I/O telemetry | \n
Laptop-only additional columns (4 fields, present in results_computer/ only):
| Column | \nDescription | \n
|---|---|
inference_time_perf_s | \nGeneration time via time.perf_counter() | \n
time_to_first_token_perf_s | \nTTFT via time.perf_counter() | \n
tokens_per_second_perf | \nThroughput computed from perf_counter timings | \n
timing_diff_ms | \nPrecision difference between time.time() and time.perf_counter() | \n
benchmarks/benchmarks_merged.csvConsolidated per-query telemetry across all three platforms. 727 rows (250 Pi 4 + 237 Pi 5 + 240 laptop) × 42 columns (41 raw columns unioned across platforms, plus a platform identifier prepended). One row per (model, platform, question); the qwen3:1.7b laptop double-run contributes 20 rows rather than 10 (see caveats).
Columns: platform (rpi4 / rpi5 / laptop), followed by all columns documented above. Pi-only columns fill as NaN on laptop rows; laptop-only *_perf columns fill as NaN on Pi rows. No values are transformed; the file is a pure union merge of the per-session CSVs with platform provenance added.
MMLU/models_MMLU_scores/{model}_MMLU.csv and .jsonOne-row-per-model aggregate MMLU results on the six-category subset used in the paper. Both formats preserved: CSV flattens per-task scores into columns (score_{task}); JSON preserves the native task_scores dict structure.
| Column | \nDescription | \n
|---|---|
model_name | \nOllama model tag | \n
overall_score | \nMean accuracy across the six categories (0–1 scale) | \n
tasks | \nComma-separated task list | \n
num_tasks | \nTask count (always 6) | \n
n_shots | \nPrompting shots (always 3) | \n
timestamp | \nRun timestamp | \n
status | \nsuccess or failure marker | \n
| \n
| \nPer-task accuracy (0–1) | \n
MMLU/MMLU_merged.csv25-row summary aggregating the per-model files above; underlies Table II's MMLU column and Table III. Columns: model, overall_score, n_shots, status, timestamp, and the six per-task score_* columns. Model tags harmonised with the benchmarks dataset — the upstream per-model file nemotron-mini_MMLU.csv appears here as nemotron-mini:4b to match the Ollama canonical form used elsewhere in the record.
rated_teaching/teaching_effectiveness_ratings/teaching_effectiveness_ratings.csvPer-response GPT-4o-mini ratings (250 rows = 25 models × 10 questions). Ratings produced via the OpenAI API; the rating prompt appears in our paper under §III-B.
\n| Column | \nDescription | \n
|---|---|
model, model_parameters | \nModel identifiers | \n
question | \nBenchmark prompt | \n
response_preview | \nFirst 200 characters of the model response (full text in the benchmark CSVs) | \n
score | \nTeaching-effectiveness rating, 1–10 scale | \n
strengths, weaknesses | \nStringified lists of rater-identified strengths and weaknesses | \n
justification | \nOne- or two-sentence rationale | \n
tokens_per_second, inference_time_s | \nCarried over from benchmark run for joint analysis | \n
error | \nPopulated when rating JSON failed to parse (score then defaults to 0) | \n
rated_teaching/human_rate_gpt4o/teaching_eval_{pseudonym}.xlsxOne workbook per rater; rater pseudonyms: bird, duck, sky, squirrel, tree. Each workbook contains three sheets:
Raters received model responses blinded via A/B/C/D labels (A=Gemma3 0.27B, B=Gemma3 1B, C=Granite4 Tiny Hybrid 7B, D=Mistral 7B). No personally identifiable data was retained; the contact email in the consent block is the principal contact above.
\nrated_teaching/human_ratings_merged.csv200-row merged dataset aggregating all five rater workbooks (5 raters × 4 models × 10 questions). One row per (rater, model, question). Models are deblinded to their canonical Ollama tags; the A/B/C/D labels visible to raters are dropped.
\n| Column | \nDescription | \n
|---|---|
rater | \nRater pseudonym: bird, duck, sky, squirrel, or tree | \n
model | \nOllama model tag (deblinded from A/B/C/D) | \n
question_num | \nQuestion index (1–10) | \n
question | \nBenchmark prompt (full text) | \n
response_preview | \nTruncated model response as shown to the rater | \n
clarity, accuracy, engagement, structure, completeness, appropriate_level, examples_analogies, actionable | \nPer-criterion rating, 1–10 scale | \n
mean_score | \nArithmetic mean across the eight criteria | \n
comments | \nRater's free-text comment, if any (otherwise NaN) | \n
annotation_analysis.ipynbJupyter notebook computing the statistics reported in paper §IV-E: Krippendorff's α, ICC(C,1) and ICC(C,k), Pearson r, mean absolute difference. Generates Figure 2.
\n| Metric | \nDescription | \nType of Metrics | \n
|---|---|---|
| Token Per Second (TPS) | \nNumber of tokens generated by the model in one second (average based on the total generation time divided by the total number of tokens). | \nHardware | \n
| Inference time | \nTotal time spent generating one output (time between the query being input to the model and the last generated token). | \nHardware | \n
| Time To First Token (TTFT) | \nTime spent to generate the first token of the output. | \nHardware | \n
| Response length | \nTotal number of characters for an output. | \nHardware | \n
| IOPS | \nInput and output memory operations. | \nHardware | \n
| Token Per Joule (TPJ) | \nNumber of tokens generated by the model per joule of energy consumed. | \nHardware | \n
| \n Massive Multitask Language Understanding (MMLU) \n | \nBenchmark used to assess the general knowledge of a model across multiple topics. | \nAccuracy | \n
| \n Teaching effectiveness \n | \nRating of the output based on teaching criteria on a scale from 1 to 10 (judged by a larger LLM: GPT-4o-mini). | \nAccuracy | \n
| Human rater | \nRating of the output based on the same eight teaching criteria on a 1–10 scale (judged by five independent human raters on a four-model subset; 200 annotations total). | \nAccuracy | \n
For full methodology, please consult paper §III. In brief:
\ngranite4:1b-h is the sole model not covered on the Pi 5. Users may analyse the broader Pi 5 dataset; the paper's three-model subset remains the analytical focus cited therein.teaching_effectiveness_ratings.csv carry score=0 and error=\"Failed to parse rating\": nemotron-mini:4b (question 9), phi4-mini-reasoning:3.8b (question 2), tinyllama:latest (question 3). Their response_preview fields contain the model output that GPT-4o-mini could not parse into the expected JSON rubric. These three are excluded from the per-model teaching-effectiveness summary used in paper Table II.nemotron-mini_MMLU.csv file records a 0% aggregate due to repeated output-format violations; see Table II's ‡ footnote. The file is retained for completeness and reproducibility of the failure mode.granite4:1b ships in BF16 (not Q4_K_M), yielding an atypical 3.3 GB on-disk footprint and 0.89 TPS — values not directly comparable to its Q4-quantised peers.granite4:1b-h and granite4:3b-h were not run on the laptop. Including this gap, full-platform coverage is: Pi 4 all 25, Pi 5 24 (missing granite4:1b-h), laptop 23 (missing both hybrid variants).qwen3:1.7b was run twice on the laptop (sessions 2025-11-19 15:26 and 15:29); both runs are preserved in results_computer/ and in benchmarks_merged.csv. Rows distinguish via timestamp. All other (model, platform) pairs have a single run.vcgencmd sampler did not fire during these runs). The question-8 row carries tokens_per_second ≈ 0.2, depressing falcon3:3b's Pi 4 throughput aggregate; exclude via response_length_chars > 0 when computing per-model means.response_preview fields. These rows inflate falcon3:3b's teaching aggregate; exclude via response_length_chars > 0 if joining with benchmarks for analysis. Combined with the three parse-failures above, the total known GPT-rating anomalies stand at five.rated_teaching/human_rate_gpt4o/annotation_analysis.ipynb at the root of this record.Members of the RSC team curate voluntarily; review timelines depend on individual availability.
\nWe aim to respond to submissions within two weeks.
\nReview criteria
\nWe check each submission against our Submission policy. Curators verify:
\nDecisions
\nSubmissions are accepted, returned for revision with actionable feedback, or declined if out of scope. Where identifying information is present, curators will request redaction before acceptance rather than decline outright.
\nConflicts of interest
\nWhere a curator has co-authored a submission, review is handled by another curator where possible.
\nAppeals
\nDepositors who disagree with a curation decision may request a second review by contacting the community via email at robotstudycompanion@gmail[dot]com
", "description": "The Robot[ic] Study Companion (RSC) is an open-source social-educational robot platform. This community curates related research artefacts, including peer-reviewed papers, datasets, code releases, theses, and supplementary materials.", "organizations": [ { "id": "edmo:3138" }, { "id": "05d8pm274" } ], "page": "The Robot[ic] Study Companion (RSC) is an open-source social-educational robot platform. The project explores how low-cost, privacy-preserving, and reproducible robotic companions can support university students' learning experiences.
\nThis Zenodo community archives research-grade artefacts produced by the RSC team and its collaborators. We aim to make the platform's scholarly output, such as papers, datasets, code, theses, and supplementary materials, more discoverable, citable, and openly accessible in one place.
\nWe accept research-grade artefacts directly supporting RSC research and development: peer-reviewed papers and preprints, datasets, code and software releases, theses, and supplementary materials tied to a clear publication or research trajectory. We do not host informal content such as slide decks, blog posts, working notes, or drafts without a defined research outcome.
\nBefore you submit
\nPlease ensure your deposit meets the following requirements. Submissions that do not meet them will be returned with feedback.
\nUpload your record to Zenodo and, on the submission form, select \"Robot[ic] Study Companion\" from the Communities dropdown. Curators review submissions on a voluntary basis, typically within two weeks depending on availability.
\nReview outcomes
\nSubmissions are accepted, returned for revision with specific feedback, or declined if out of scope. Curators may request redaction of identifying information as a condition of acceptance. Declined submissions may be resubmitted once the issues are addressed.
\nEach record in this community has its own DOI and should be cited individually. If you additionally wish to acknowledge the RSC platform itself, for example, when your work builds on the RSC or deposits into this community, we suggest the following.
\nAcknowledgement line
\n\n\nArtefact deposited in the Robot[ic] Study Companion community on Zenodo (https://zenodo.org/communities/rsc).
\n
Canonical RSC reference
\n\n\nBaksh, F., Zorec, M. B., & Kruusamäe, K. (2024). Open-Source Robotic Study Companion with Multimodal Human–Robot Interaction to Improve the Learning Experience of University Students. Applied Sciences, 14(13).
\n
We recommend citing this publication when referencing the RSC platform in publications and derived works.
\nFor questions about the community or to discuss a potential contribution, visit rsc.ee and fill out this quick form \ud83d\udcc4 or email us at robotstudycompanion@gmail[dot]com
", "title": "Robot[ic] Study Companion", "type": { "id": "project" }, "website": "https://rsc.ee/" }, "revision_id": 16, "slug": "rsc", "updated": "2026-04-17T14:55:26.698529+00:00" } ], "ids": [ "4a41b11b-322a-4d30-b777-7b7ab3a3c527" ] }, "id": "19643020", "pids": { "doi": { "client": "datacite", "identifier": "10.5281/zenodo.19643020", "provider": "datacite" } } }, "pids": { "doi": { "client": "datacite", "identifier": "10.5281/zenodo.19643021", "provider": "datacite" }, "oai": { "identifier": "oai:zenodo.org:19643021", "provider": "oai" } }, "revision_id": 14, "stats": { "all_versions": { "data_volume": 16819016.0, "downloads": 8, "unique_downloads": 7, "unique_views": 53, "views": 77 }, "this_version": { "data_volume": 16819016.0, "downloads": 8, "unique_downloads": 7, "unique_views": 53, "views": 77 } }, "status": "published", "swh": {}, "updated": "2026-04-21T14:00:28.349920+00:00", "versions": { "index": 1, "is_latest": true } }