I Built an MCP for My TV — and What It Taught Me

I built an MCP server that turns Claude into the remote for my living-room TV: one local server, 14 tools, talking straight to an LG webOS TV on the home network — power, volume, apps, channels, typing, and screen capture so the agent can see what is on screen. The interesting part is not the code. Once the connection worked, what was still missing was not engineering. It was teaching.

This is an edge optimization on a process that takes no more than five minutes a day, and I know it. But every futuristic movie starts with the hero telling the smart home “put on the news” and it happens. So the future is here, and it runs over a websocket on the home LAN.

The principle: AIOS, and who is responsible for what

Let’s start getting used to the name AIOS — AI Operating System. That is what the Claude and GPT desktop apps really are: an environment with wide permissions over your machine, holding general-purpose agents that plan, execute, and correct themselves.

That gives a clean division of labor, the core of every adoption project I run:

• The technologist exposes the data and tools inside a connector — MCP, plugin, extension, the name matters less — that is stable and safe, and installs it.
• The users, including non-technical ones, learn to write their own skills: plain-language instruction pages that teach the agent how to run a process.

The TV is a small, clean example of both halves, seam included.

What I actually built

LG TVs speak SSAP — JSON over a websocket at wss://<ip>:3001, behind a self-signed certificate. I started from the open-source work around webOS but ended up writing my own client on top of ws: the common library, lgtv2, is stale, callback-based, and drags in a legacy websocket package. What I did take from it was the known-good permission manifest used for pairing. Three npm dependencies in total: the MCP SDK, ws, and zod.

Decisions that look technical but are really product decisions:

• Pairing happens once. The first control call makes the TV show an approval dialog. Accept it with the physical remote, the TV issues a client key stored locally, and it never asks again. The key works only inside the LAN, only against that TV, and holds no account credentials.
• get_status touches nothing. It reports power, pairing state, current app, and volume, and it works before pairing too, without triggering the dialog.
• Turning the TV on needs Wake-on-LAN. A TV in standby closes every TCP port, so power_on sends a magic packet over UDP instead and waits for it to boot, roughly 15 seconds.
• Button presses skip the main socket. You ask the TV for getPointerInputSocket, get a one-time wss URL, and write line-oriented messages to it.
• The address changes, and that is fine. If the last-known IP stops answering, the server resolves the TV over mDNS and updates its config.

The tool that was missing: feedback

The problem I hit is that this ecosystem performs actions but cannot tell you what is on the screen. You can press right; you cannot find out what happened next. For an agent that is fatal — it navigates blind.

So I borrowed the idea from another open-source project — a single screen-capture tool — and implemented it here. It is an executeOneShot call producing a 960×540 JPEG, which the server downloads from the TV and returns as an image block. DRM-protected video comes back black, but app UI and menus render fine — so a black frame is itself a signal that something is playing.

That is the most significant change in the project, and it was not in the original plan: the moment the agent can look, it can correct itself. This generalizes well beyond a TV. A tool that only acts and never reports state forces the user to be the agent’s eyes, which is exactly where automation breaks. When I build a connector for a client, the first question is not “which actions” but “how will it know the action worked”.

What the server deliberately does not do

A lot of the work in a good connector is in the limits:

• press_button caps out at 10 repeats, with 150 milliseconds between presses. No infinite navigation loops.
• No retry storms: at most one reconnect per tool call, and never a pairing attempt in a loop.
• The skill shipped alongside the server tells the agent to prefer launch_app over navigating menus with arrow keys.
• Deliberately left out: LG’s cloud API, multi-TV support, HDMI-CEC, and mouse-pointer movement.

The suite runs against an in-process fake TV — and a fake mic for the sound tools — so it verifies with no hardware.

Two smaller calls I like: error messages are in Hebrew, because that is the household this server serves. And after the basics worked I added two tools that measure room loudness through the laptop microphone (they shell out to ffmpeg, and the capture path is macOS-only), plus a one-time calibration against a phone sound-meter app, so the agent can close the loop itself. I have not run that calibration yet — until I do, the reading is a rough estimate, not a real dB figure.

The server is plain local stdio, so the same code drops into any MCP client — I run it in Claude Code and in ChatGPT Work, where it is also packaged as a plugin. One connector, several environments.

Where the demo stalled

That is where my job as the technologist ends: the agent got new capabilities, and with them it can do “anything” on the TV too.

But the demo I recorded shows the process was not finished. The MCP works, but the latency is terrible, and the agent could not find its way to what it was looking for inside the yes+ app — yes+ being the streaming app of yes, one of Israel’s pay-TV providers.

That gap is the point. Not a bug, not a missing tool. Nobody had taught the agent how to navigate a TV, the way you teach a new human user: where to start, where search lives, when to screenshot and check before pressing anything else.

That skill, if and when someone writes it, will not look like code. It will look like a paragraph: “to reach a show in yes+, open the app, wait for it to load, screenshot to confirm you are on the home screen, and only then start moving.” The best person to write it is not me — it is whoever navigates that app every evening.

What this means in a business

Teaching Claude to navigate inside yes+ is not a technology task. It is a capability everyone who works at a computer should have. That is what AI adoption means.

Why learn it? Because it is fun and easy, and because once the barrier breaks and the cost of building a skill approaches zero, creativity gets unlocked — useful things get built bottom-up, by the people who know the process.

The process I run with organizations is three steps: you buy a Claude or GPT subscription; I break every barrier and connect it to everything, safely and stably; and I teach you and your staff to use it.

The third step is the longest, and it does not end. A connector gets installed once; skills get written every week, by different people, for processes nobody would have specified in advance. That is the part I cannot do for you.

The takeaway

The engineering half of AI adoption is smaller than it looks and finishes early. An MCP server for a TV is two evenings of work; teaching an organization to navigate with it decides whether anything changes. Measure adoption by connectors installed and you are measuring the easy half.

If it is the other half you care about, here is how the process works.