VLM Action Parser Library

Module for prediction and execution of robotic skills using vision-language models (VLMs).

Initial textual instructions (e.g. task board completion steps) along with an optional auxiliary image (e.g. depicting taskboard components) are processed into a robot-executable task list. This module relies on a skill library (consisting of motion primitives for executing tasks , e.g. steps in taskboard benchmark).

It can also be queried to determine action success (e.g. whether or not the door has been opened). 

Internally, it uses langchain, so the module can connect to different VLMs (local models or OpenAI API).