Wednesday, July 22, 2026

The Lawyer I Never Hired: How ChatGPT Got Me $4,760 Out of Norse Atlantic

TL;DR: Our Norse Atlantic flight out of Oslo was delayed overnight for technical reasons. Norse offered a $25 refreshment card and then ignored our claims. Because the flight departed the EEA, EU261 applied even though we live in New York. I had no idea how to enforce a European regulation against a Norwegian company from 6,000 kilometers away. ChatGPT-Pro did. It walked me through the national complaint body, the Norwegian conciliation court, the service-of-documents problem, and a settlement negotiation down to the last øre of interest, one step at a time. Eleven months later, Norse wired $4,760.36. This is one passenger's account, not legal advice.




In August 2025, my family and I were booked on Norse Atlantic flight N0 01 from Oslo (OSL) to New York (JFK). Scheduled departure 20:05, scheduled arrival in New York 22:00, same day.

On the morning of the flight, Norse emailed us that departure was pushed to 09:00 the next morning. Technical reasons. Overnight delay. Four of us, one night in Oslo.

To soften the blow, Norse offered a $25 digital Mastercard for refreshments.

Twenty-five dollars. For a family of four. For an overnight delay at an airport hotel.

We did not actually mind the delay. An extra night in Oslo, a proper dinner, and a flight to New York taken after a full night of sleep instead of as a red-eye is not much of a hardship. What stood out was the twenty-five dollars. Not the wait. The number.

I did not know the first thing about what we were owed, or whether a European regulation even reached a country that is not in the EU. I did not go digging through statutes I would not have understood. I pasted the delay email into ChatGPT-Pro and asked it what our rights were.

That question started an eleven-month correspondence in which ChatGPT became my lawyer.


First thing my lawyer told me: Norway does not matter the way I thought.

My assumption, and I suspect most people's, was that EU passenger-rights rules do not apply to Norway, because Norway is not in the EU. That assumption would have ended the case on day one.

ChatGPT corrected it in the first reply. Norway is not in the EU, but it is in the EEA, and EU Regulation 261/2004 applies to flights departing an EEA airport, regardless of the airline's nationality or the passenger's residence. A flight out of Oslo Gardermoen is covered exactly like a flight out of Frankfurt. When I pushed back, twice, because I did not believe it, it walked me to the EU's own passenger-rights page confirming that the rules extend to Norway, Iceland, and Switzerland.

Then it drew a distinction I would never have drawn on my own, and that distinction turned out to be the whole game.

There is the right to care. During an overnight delay, the airline must cover meals in proportion to the wait, a hotel, and transport to and from it. If they do not provide it and you pay, you keep receipts and claim it back. A $25 card does not come close to discharging that.

And there is standard compensation. For a long-haul arrival delayed three hours or more, not caused by genuinely extraordinary circumstances, you are owed a fixed sum. For our route, 600 euros per passenger. Four passengers. 2,400 euros.

A technical fault, ChatGPT noted, is almost never an extraordinary circumstance, and the airline carries the burden of proving that it was. I filed that sentence away without understanding yet how much work it would do.


It drafted the claims. Norse ignored them anyway.

ChatGPT wrote both claims, told me to file them as two separate things rather than one, and told me which articles to cite in each. On August 18, 2025, I filed the standard-compensation claim. Norse acknowledged it and quoted six to eight weeks. On August 23, I filed the care-expenses claim with the receipts, hotel at 4,534.80 NOK, dinner at 3,409 NOK. Norse acknowledged that one too.

And then nothing. Six weeks, eight weeks, nine weeks. No decision, no denial, no human.

This is the part the airline is counting on. The refreshment card is not really an apology. It is a filter. Pay a tiny sum, ignore everyone who asks for more, and settle only with the fraction who refuse to disappear. Left to myself, I would have been in the group that disappeared. I did not know there was anywhere else to go.

My lawyer knew.


It knew which door to knock on next.

When an airline ignores you and the flight touched Norway, ChatGPT explained, the next stop is Transportklagenemnda (easy to remember, rolls right off the tongue, a bit like Ip-ei-ro-tis), the Norwegian Air Passenger Complaint Handling Body. I had never heard of it. It told me what it was, what it could do, what to upload, and what to write, then drafted the complaint. It even walked me through the Norwegian power-of-attorney forms field by field, told me I did not need separate ones for my two children since I am their legal guardian, and showed me how to organize the evidence into a clean set of files. The case registered as 2025-04062.

It also predicted, correctly, that Norse's silence would work in our favor. Transportklagenemnda asked Norse to prove extraordinary circumstances. Norse did not answer. Unchallenged claims are then normally treated as correct. By ignoring the process, Norse forfeited the only defense it had, exactly as ChatGPT had said it would.

On April 23, 2026, the secretariat issued its decision. No extraordinary circumstances. It recommended that Norse pay 2,400 euros in standard compensation and 6,240 NOK in care expenses.

We had won, in writing, from an official body. I let myself feel finished for about a day.


Then it told me the thing I really did not want to hear: Nobody cares about the Transportklagenemnda 

I sent the decision to ChatGPT half expecting congratulations. Instead it told me the decision was a recommendation. A recommendation is not a binding order. Transportklagenemnda decides who is in the right, but it has no power to make the airline pay, and an airline that chooses to ignore it faces no penalty for doing so. (Apparently, a common theme across all EU countries, regarding EU261.)  It has no teeth. Left to myself, I would have forwarded that beautiful PDF to Norse and sat waiting for a check that was never going to arrive.

On its instruction, I emailed Norse the decision with a fourteen-day deadline. Norse did nothing, again.


And then it told me to sue: Forliksråd time!

This was the fork where, on my own, I would have quietly run out of road. I would have sent Norse another annoyed email, maybe complained again to Transportklagenemnda, and eventually let the whole thing go, because the alternative sounded absurd. Sue a Norwegian airline, in Norway, from a couch in New York? That is not a thing I thought normal people did.

ChatGPT told me it was, in fact, a thing normal people do. It explained that the recommendation was now ammunition rather than an ending, that the next stop was the Forliksråd, the Norwegian conciliation court, the one venue that can produce an enforceable judgment, and that pursuing it from abroad was routine rather than exotic. Filing a small claim against a company 6,000 kilometers away was not the ordeal I had imagined. It was mostly paperwork, and paperwork was the part it was built for.

So the case stopped being about airlines and started being about Norwegian civil procedure. That is precisely the terrain where someone like me gets lost.


Filing from New York.

ChatGPT flagged the first obstacle before I hit it. The Norwegian digital court system runs on BankID, the national electronic ID, which I cannot get as a US resident. The online path was closed to me, so I would have to file the slow way, by email and on paper.

Then it caught another trap. My instinct was to file in Oslo. You file where the defendant is registered, and Norse Atlantic Airways AS is registered in Arendal, organization number 926 493 647, at Fløyveien 14. The clean venue was Arendal og Froland Forliksråd, under Namsfogden i Agder. Filing in Oslo would have bounced the case and cost weeks.

On ChatGPT's advice, I filed the forliksklage twice: once by email to the Agder police district (officially not a supported way to file, but it opened an email channel with them), once as a tracked paper copy mailed from New York, in case the email alone was not treated as a formal filing. The packet crossed the Atlantic, sat in customs for a while, and was delivered on June 1, 2026. The case became F2026-014542.

It also told me to keep the claim clean: mirror the Transportklagenemnda decision exactly, 2,400 euros and 6,240 NOK, plus the filing fee and statutory interest, and drop everything else. A claim that matches the official decision is far harder to argue with than a padded one. On my own, I would have thrown in everything and weakened the whole thing.


Faste møtefullmektiger

The paper copy landed in Norway, and then the real complication surfaced. Because I live in the United States, the court would eventually have to serve documents on me abroad, which can trigger formal international channels and translation, meaning cost and months of delay. The case officer confirmed it and suggested I appoint a representative, offering the name of a well-known claims agency as an example. That agency wanted 35 percent, going to 50 percent if legal action is needed (which was the case for me). On a recovery of roughly 4,000 dollars, that is about 2,000 dollars, handed to a company for logistics, on a case that was already decided.

The agency was not the only option, and ChatGPT knew it. It told me that Norwegian conciliation courts keep a regulated roster of faste møtefullmektiger, fixed meeting representatives, who can receive documents and appear on your behalf on a narrow, limited basis, for a fee capped by regulation at a small fraction of a court fee. Almost nobody outside Norway knows this exists. I certainly did not. It drafted the request for the list, drafted the individual outreach emails, and drafted the limited authorization so that the representative could receive mail and stand in a room but could not settle, waive, or reduce the claim without me.

Getting one of them to say yes turned out to be the hard part. I wrote to several people on the list, one at a time, and several said no. Norway is a small country. These are local people in small towns, and more than one of them did not want to attach their name to a case against a large airline they might cross paths with again. What looked like routine paperwork to me looked, to them, like picking a fight with a neighbor.

Then Stein Hindar said yes. Fixed fee, 2,690 NOK, about 270 dollars, no percentage of anything. He was straightforward, quick to answer, and willing to act where others hesitated. His real value to the case was a Norwegian mailing address, which made the international-service problem mostly evaporate. Two hundred seventy dollars, fixed, instead of two thousand, contingent. If you are outside Norway and end up walking this same path, he can be reached at [email protected], and after how this went, I am glad to send business his way.


Norse woke up, and my lawyer read the fine print.

The pattern is worth stating plainly. Norse ignored the airline claim. Norse ignored Transportklagenemnda. Norse ignored my post-decision demand. What Norse did not ignore was a live case in a Norwegian court that could produce an enforceable judgment against a Norwegian company with Norwegian bank accounts. 

Suddenly, a human from Norse wanted to settle.

The first offer was not terrible (2400 euros for standard compensation, plus ~5000 NOK for hotel and dinner), and part of me wanted to take it and be done. ChatGPT talked me out of that. With the case sitting in a Norwegian court, it said, this was not the moment to be nice. Norse had left three things on the table, and I should demand every one of them in full.

The dinner: Norse had substituted its own idea of a "reasonable" amount. I should insist on the full figure Transportklagenemnda had already approved, not the discounted version. The legal costs: full reimbursement of both the filing fee and the representative fee, since Norse had forced the filing by ignoring everything up to that point. And the interest: statutory late-payment interest, calculated to the day, for every day Norse had sat on the money since the previous Fall. Go for the jugular.

So it answered with numbers instead of adjectives. Norwegian statutory late-payment interest is set by the Ministry of Finance every six months: 12.25 percent through the end of 2025, 12.00 percent through mid-2026, 12.25 percent after. Applied to the award from thirty days after each original claim, it came to roughly 233 euros and 595 NOK by early July, accruing a little under a dollar a day after that. ChatGPT gave me the figures, the start dates, and the daily accrual, so there was nothing left to wave away.

Then it made another move in the entire saga, and it is the one I would never have made. When Norse agreed to the principal and costs, it went quiet on interest; my instinct was to accept and be done. ChatGPT stopped me: No, we go for the interest as well! Let's get what they offer first, and then continue.

But accepting the offer as written risked being treated as full and final settlement, quietly waiving the interest we had just calculated. It wrote the paragraph instead: I would send banking details, accept the listed amounts as partial payment, expressly reserve the interest, and withdraw the case only after the full amount, interest included, had cleared. A krone short, and I would see Norse in court in a few weeks. Well, Stein would see them, but close enough. That one paragraph is the difference between closing a case and giving a piece of it away.

Norse folded on the rest. The final queue included the compensation, the full care expenses, the filing fee, the representative fee, and the interest.


$4,760.36.

On July 14, 2026, the wire landed. 4,760.36 dollars.

The 25-dollar refreshment card had become a 4,760-dollar wire. It took eleven months, a national complaint body, a conciliation court, a local representative, a stack of PDFs, and a great deal of precision, almost none of which I supplied myself.

None of this soured me on the airline. I like Norse, I will fly them again, and I recommend them: their "business" class is some of the best value for money I have found for flying the family to and from Europe. A good airline simply ran a claims process built to make people give up, and I happened not to.


The lawyer I never hired.

ChatGPT-Pro did not win the case; the merits and Norse's own silence did that. And it did not take decisions out of my hands. Every send button was mine, and once or twice I overruled it.

What it did was handle a dispute that was not complex on the facts but brutal on procedure, in a foreign language, across a border, over eleven months. That is precisely the kind of problem that used to require a lawyer and, it turned out, no longer did.

It read the regulation I did not understand. It knew the complaint body I had never heard of. It told me a favorable decision was not a check, so I did not wait on one. It diagnosed the BankID and service-of-documents problem before the court raised it. It knew about the regulated fixed-fee representative and steered me off the 35-percent agency, which is worth well over a thousand dollars by itself. It fixed my venue from Oslo to Arendal before I misfiled. It did the interest arithmetic to the øre. And it caught the settlement trap that would have cost me the interest at the finish line.

Above all, it held the entire case in memory. Dates, amounts, case numbers, deadlines, who said what and when, which document proved which fact. Across dozens of sessions it was the one participant that never lost the thread. A human lawyer keeping that continuity would have billed for every hour. Here it was the default behavior of a chat window.

I did not hire a lawyer. I described my problem, uploaded my documents, and did what my lawyer told me to do.


If you are a Norse passenger stuck outside Norway.

The specifics are Norse and Norway, but the shape holds for any EEA-departing flight. A few things worth knowing before you start.

Save everything, immediately. Booking confirmation, delay notice, boarding passes, every receipt, every acknowledgment email. The case is won on documents, and the airline is hoping you did not keep them.

Your residence does not matter. The departure airport does. If you left the EEA, EU261 covers you, whether you live in New York or Nairobi.

Compensation and care expenses are two different claims. File them separately and cite the right articles for each.

A complaint-body decision is not payment. It is powerful evidence and not a cent of actual money. If the airline still stalls, the recommendation is your ammunition, not your finish line.

Do not hand a claims agency 35 percent. For a Norwegian venue, ask the conciliation court for its list of fixed meeting representatives. A regulated, fixed-fee local representative solves the service-of-documents problem for a fraction of what a percentage agency takes. Stein Hinder was quick and easy to work with, if you ever need one:  [email protected]

File where the company is registered. For Norse Atlantic Airways AS, that is Arendal, not Oslo. The wrong venue costs weeks. You can submit unofficially by email, but you also need to send everything on paper, or ask your representative to do so.

Claim your costs and interest, with exact numbers. Filing fee, representative fee, and statutory late-payment interest are all recoverable. Precise figures and start dates leave nothing to argue about.

Never accept as full and final until the money clears. If you take a partial payment, say in writing that it is partial and that you reserve the rest. Withdraw nothing until the wire arrives.


What actually changed.

I think this saga is an early signal of something larger.

For a claim worth a few thousand dollars, hiring a lawyer never made sense. The fee would have swallowed the award, and finding a lawyer who actually knew EU261, Norwegian conciliation procedure, and cross-border service would have been close to impossible. That is exactly why the agencies that specialize in this keep 35 to 50 percent: the expertise is rare, the amounts are small, and the arithmetic only works if they take a large cut. AI collapsed that arithmetic. I got the specialized help without the specialist, and without signing away half the recovery to get it.


And what did not.

It would be a mistake, though, to read this as AI making justice fast or frictionless. The knowledge bottleneck fell away. The rest did not. The case still took eleven months, and almost none of that was drafting time. It was waiting: for a complaint body to work through its queue, for an airline to let a deadline pass, for a paper packet to cross an ocean, for a court to register a filing. A lawyer would have waited exactly as long. Those delays are structural, and no model removes them.

That gap is the subject of a recent Lawfare report by Justin Curl, Sayash Kapoor, and Arvind Narayanan, AI Won't Automatically Make Legal Services Cheaper. Their argument is that AI drives down the cost of legal outputs, the drafting and the research, while the things that actually gate outcomes are structural: professional regulation, adversarial incentives, and the speed of the humans who still have to decide. Cheaper drafting does not touch any of them. Their subject is the American system, but the logic traveled to mine. The part AI could reach, it transformed. The part it could not reach still took a year.


Why it will not fix itself.

None of this was automatic either, which is the report's sharper warning. The door opened for me because Norway lets a foreigner self-represent and hire a cheap local stand-in for the parts that need one, and because no one treated my chatbot as practicing law without a license. I used the AI on my own case, as any pro se litigant can. Change those conditions and the door swings shut again.

The obvious next question is whether an agent could run the whole thing end to end while I went to dinner. Not soon, and perhaps not ever, because the parts that remain are human-paced, and they do not expose an API. Worse, the airline uses that delay on purpose, and the parties who benefit from a slow system are the last ones with any reason to speed it up. Norse ignored me for months because ignoring people works. An agent does not change that math. Armed with the best AI in the world or with nothing at all, a claimant hits the same wall. AI lowered my cost of refusing to give up. It did nothing to Norse's ability to wait me out, and every company watching has read the same playbook.


One last thought.

I turned a $25 snack voucher into a $4,760 wire. It only took eleven months, a Norwegian conciliation court, and more stubbornness than any reasonable person would spend on the principle. That is the trick, and it is also the catch. Norse offers the twenty-five dollars because it is betting you have a job, a life, and better things to do than wage a transatlantic paperwork campaign. Usually it is right. This time it drew the one guy in New York with a ChatGPT Pro subscription, a grudge he refuses to call a grudge, and nothing better to do. 

So here is my advice, which is not legal advice, because I am not a lawyer and neither, technically, is my lawyer: keep the receipts, fire up that chat, and let a chat window turn you into Alan Shore (or Denny Crane, if you prefer; I do not judge).

Revenge is a dish served cold, like smoked Norwegian salmon on a toasted NYC bagel.

Saturday, March 28, 2026

Taste Is Not Enough. Reality or Bust


A paper published in Nature on March 25, 2026 describes "The AI Scientist," a system built by Sakana AI that automates the full cycle of scientific research: idea generation, experiments, analysis, writeup, even peer review submission.

The marginal cost of producing paper-shaped research output is collapsing.

So, when paper production becomes cheap, what is next?


100,000 axiom systems and counting

Stephen Wolfram spent years systematically enumerating all possible axiom systems. Each axiom system defines a "possible universe of mathematics": a different set of starting rules, a different universe of theorems. Most universes are empty or trivial. Of the ones that are not trivial, many are bizarre. Our entire familiar mathematics occupies a tiny corner of this space. Logic, specifically Boolean algebra, turns out to be perhaps the hundred thousandth axiom system you would encounter if you enumerated them by complexity.

Wolfram found nothing obviously special about the axiom systems we actually use. His suspicion (not a proof) is that we study them for largely historical reasons: they are generalizations of arithmetic and geometry from ancient Babylon. The space of possible mathematics is vast, our explored corner is small, and we are in this particular corner because of history, not because of any intrinsic property of the systems themselves. 

Why do we use them? Because humans decided they were interesting.


The AI Scientist has the same problem, one level up

The AI Scientist can generate research ideas, execute experiments, and write papers. Sakana reports a cost of roughly $15 per paper. One of three papers passed peer review at a ICLR workshop (not the main conference track), with humans filtering the most promising outputs before submission. The system can produce formally structured research outputs. It cannot yet tell which ones matter.

The problem is not just quality filtering. A separate study in Nature earlier this year analyzed 41.3 million research papers and found that scientists using AI tools publish three times more papers and get five times more citations. Great for individuals. But collectively, AI-driven research covers less topical territory. It clusters around already-popular problems. 

In the Wolfram analogy: a machine that evaluates "interesting" by pattern-matching against known mathematics will keep steering you back to the hundred thousandth axiom system and its neighbors. Lots of exploit, much less explore.


So what is the actual scarce resource?

This matches my own experience using AI agents for research and teaching. The agents are shockingly good at execution. Give them a clear task with well-defined scope and they deliver something genuinely useful, fast.

But "work on the next most important task" only works if someone figured out what the important tasks are. The agent does not decide which questions matter. The moment you ask it to define its own scope, you get the AI Scientist problem: lots of output, most of it predictable, much of it wrong in ways that require domain expertise to even detect.

The scarce resource is judgment. The ability to look at a vast space of possibilities and say: this one.

That is the comforting answer, anyway. AI does the grunt work. We provide the taste, the direction, the vision. We stay at the center of the universe.

Except: that story is cope. Rich Sutton's "Bitter Lesson" showed that every time researchers tried to hand-code human knowledge into AI systems (chess heuristics, vision algorithms, Go strategies), brute-force scaling eventually crushed the hand-coded approach. Human judgment about what matters may just be the next ontology in line to be bypassed. But even if it is not, history suggests it was never as reliable as we like to think.


But the world has a vote

Wolfram's enumeration is purely abstract. The axiom systems just sit there, inert. But science interacts with data from the world we observe. You hypothesize, you collect data, and reality tells you whether you are wrong. And that feedback loop has a history of promoting "useless" systems to central importance, often over the explicit objections of the people who understood them best.

Godfrey Hardy, a godfather of number theory, wrote in 1940 that number theory had a kind of supreme uselessness, that no one had discovered any warlike or practical purpose for it, and it seemed unlikely anyone ever would. And he was proud of that uselessness, as a sign of the supreme taste of a pure mathematician. 

Thirty-one years after his death, RSA encryption arrived, and modern cryptography now depends heavily on the number theory Hardy was so proud to call pointless.

Maxwell predicted electromagnetic waves in 1865 as a mathematical consequence of his equations. Hertz demonstrated them physically in 1887, and when his students asked what the discovery was good for, he replied: "It is of no use whatsoever. This is just an experiment that proves Maestro Maxwell was right." 

Marconi built the wireless telegraph less than a decade later.

Notice what Hardy and Hertz have in common. They were not amateurs. They understood their own discoveries better than anyone alive. Their taste was extraordinary: out of the vast space of possible mathematics and physics, they picked systems that turned out to be profoundly important. But their forecasts of usefulness were completely wrong. Hardy looked at number theory and said: this is beautiful and this is deep. He was right about that. He was wrong about what the world would do with it. Hertz looked at electromagnetic waves and saw a confirmation of Maxwell. He was right about that too. He could not see the wireless telegraph.

The distinction matters. Taste selected the right systems to study. But taste could not predict what those systems would be for. That was decided later, by technologies and applications that did not yet exist. The world retroactively decided which "useless" formal systems had been important all along.

So the comforting story ("AI does execution, we provide the visionary taste, we stay at the center of the universe") is incomplete. Taste is real but taste without reality is flying blind. Entire fields operate this way: elegant theory frameworks that survive for decades because they never invite reality to correct them. And the people with the best taste in history still could not see where their work would land.

The right question is not "who has the best taste?" It is "what kind of feedback loop lets reality surface the value that taste alone cannot see?"


Can AI close that loop?

In some fields, it already has. 20 years ago, Mechanical Turk returned human judgements through API calls. Now, autonomous wet labs (Emerald Cloud Lab, Strateos, RAPID-200) accept experimental protocols via API and return physical results without human hands touching anything. An AI agent can already design an experiment, submit it to a cloud lab, and get data back. The loop with physical reality is not a future idea. It is existing infrastructure.

And still, the narrowing problem persists. Ten thousand automated experiments over a weekend still require someone (or something) to decide what experiments to run. The labs automate verification, not direction. Reality is the slowest, most expensive API there is. A clinical trial takes years. Growing a test crop takes a season. AI generates hypotheses at near-zero marginal cost, but verifying them against the physical world still costs capital and time.

So, the question is what happens when AI-generated ideas start getting corrected by the world. That is the difference between an AI that enumerates the space of possible mathematics and one that discovers non-Euclidean geometry because spacetime forced its hand.

Tuesday, March 17, 2026

How I Stopped Being a Copy-Paster for My AI Agent: Claude Code, Google Cloud, and the Loop to Close

TL;DR: Your AI agent in Claude Code on the Web can use Google Cloud (or AWS/Azure) to store large datasets, run long computations, deploy web apps, and schedule recurring jobs. Once you have a cloud account and project, the repo-specific setup takes about five minutes:

  1. Set an encryption password in your environment settings (see Step 1 below). If you only use one cloud provider, name it CLOUD_CREDENTIALS_KEY. For provider-specific setups, use GCP_CREDENTIALS_KEY / AWS_CREDENTIALS_KEY / AZURE_CREDENTIALS_KEY.
  2. Tell the agent: "Install the cloud-bootstrap skill from https://github.com/ipeirotis/cloud-bootstrap into this repo."
  3. Tell the agent: "Set up GCP access for this project."

The agent walks you through the rest, including one command you run in Cloud Shell to generate a temporary token.



The moment I became a human copy-paster

A few weeks ago, I was debugging data issues on the mturk-tracker demographics site. Claude Code would write a diagnostic script. I would deploy it to the server. I would copy the output. I would paste it back into Claude. Claude would write the next script. I would deploy that one. Copy. Paste. Deploy. Copy. Paste. Deploy.

I was not managing an AI agent. I was its copy-paster. Claude did the thinking. I did the Ctrl-C, Ctrl-V.

That was problem number one.

Problem number two: I needed to collect data from several websites, a process that would take a day or two of continuous scraping. Claude started the work, but the sandbox kept timing out. The session would die, I would restart it, Claude would pick up where it left off, and then the session would die again. The only way to keep things moving was to babysit: Break the bigger task into smaller subtasks and then "Do next task." "Do next task." "Do next task." Over and over. I was not reviewing or directing anything. I was just pressing the button to keep the machine running. I understand that this is our new role as humans, serving our new AI overlords, but... boooooring.

Problem number three: I needed to train a model that required a GPU. The Claude Code sandbox does not have GPUs. So I had to manually launch a VM on Google Cloud, SSH into it, clone the repo, install the dependencies, start the training, and then remember to check back later and shut the machine down before it burned through my budget. Claude had written all the training code. But the last mile (getting it to actually run somewhere with the right hardware) was entirely on me. The AI writes the code. The GPU does the math. And I am the guy who forgets to shut down the machine. Guess which component has the highest error rate.

Three different problems. Same root cause. The sandbox is a walled garden. Claude can think, it can code, it can analyze. But it cannot reach the outside world. It cannot talk to a server, run something overnight, or spin up a machine with a GPU. Everything that requires infrastructure beyond a small ephemeral container? That is your job.

The fix: give the agent a cloud account.


What changes once the agent has cloud access

Remember the mturk-tracker debugging? With cloud access, Claude deploys its own diagnostic scripts to Cloud Functions, runs them against the live data, reads the results, and iterates. No copying. No pasting. No human in the middle.

The web scraping that required me to babysit? Claude deploys the scraper as a Cloud Function with a scheduler. It runs every 15 minutes, stores results in a Cloud Storage bucket, and I check in the next day. I literally went to sleep and woke up with the data collected.

The GPU training? Claude launches a VM with the right specs (say, an n1-standard-4 with a T4 GPU), clones the repo, installs everything, starts training, and sets up a shutdown script that kills the machine when the job finishes. Results go to Cloud Storage. I went to dinner. When I came back, the model was trained, the results were in the bucket, and the VM was already off. The alternative was me manually SSH-ing into a machine, running htop every twenty minutes, and hoping I remembered to shut it down before I went to bed. (Ask me how I know that "hoping I remember to shut it down" is not a reliable cost management strategy.)


The setup (yes, there is some setup)

I will walk through this using Google Cloud, since that is what I use (the concepts are the same for AWS and Azure). If you do not already have a Google Cloud account, go to cloud.google.com and sign up.

Once you have an account, create a project in the Cloud Console. A project is Google Cloud's way of organizing resources and billing. Click the project dropdown at the top, click "New Project," give it a name, and note the project ID.

You do not need to install anything on your own computer. When you need to generate a token, you will use Google Cloud Shell: a browser-based terminal with everything pre-installed.


My pattern: one repo, one cloud project, same name

Every GitHub repo I work with gets its own dedicated Google Cloud project. And they get the same name. The repo paper-oral-exams gets the Cloud project paper-oral-exams. The repo course-ai-pm gets the Cloud project course-ai-pm.

Why? Mostly resource isolation. The agent for the course repo cannot accidentally touch the research data. Each agent gets exactly the access it needs for its own project and nothing else. It also makes housekeeping easier: when everything for a project lives in one Cloud project, you can quickly spot which storage buckets, databases, and VMs are still needed and which are leftovers. No more "wait, whose VM is this and why is it still running?"

Creating a Cloud project is free and takes 30 seconds.


Service accounts: giving the agent its own keys (not yours)

When you use Google Cloud, you log in with your Google account. But an AI agent is not you. And more importantly, it should not be you. Your Google account has access to everything: your email, your billing, your entire cloud infrastructure. Giving all of that to an automated tool would be like handing your intern the keys to the building, your credit card, and your Netflix password. Just in case.

Instead, you give the agent a service account: a restricted identity designed specifically for automated tools. It has its own email address (something like [email protected]) and you decide exactly what it can do. Read from this storage bucket. Deploy this function. Query this database. Nothing more.

A caveat: the approach below (encrypting a service account key in the repo) is a pragmatic workaround for agent environments that do not yet support proper workload identity or secret stores. If the worst case is 'the agent ran up a $200 bill on a research project,' you are fine. If the worst case involves production data or your personal credentials, use something else. When proper agent identity federation exists, this will get simpler. For now, it is the best approximation available.

The service account authenticates using a key file: a JSON file that acts as its password. Whoever has this file can act as the service account. Which means this file needs to be protected.

But here is the catch: Claude Code runs in a sandbox that resets after each session. The only thing that persists is the GitHub repo. So the key file needs to live in the repo somehow, but committing a plaintext credentials file to a repo is a classic security mistake. (It is so common that GitHub literally has automated scanning to catch people doing it.)

The solution: encrypt the key file and commit the encrypted version. The encryption password lives in an environment variable in Claude Code, which persists across sessions but never enters the repo. At the start of each session, a hook decrypts, authenticates, and deletes the plaintext immediately. The encrypted file is useless without the password. The password is useless without the encrypted file. And if in the worst case scenario your password leaks, you only exposed the service account with limited permissions, and you can always deprecate and regenerate the credentials. 


The five-minute walkthrough

You do this once per repo.

Step 1: Set your encryption password.

In Claude Code, open the environment settings for your session and find the "Environment Variables" field. Add a new variable:

CLOUD_CREDENTIALS_KEY=your-strong-passphrase-here

(If you work with multiple cloud providers across different repos, you can use provider-specific names like GCP_CREDENTIALS_KEY or AWS_CREDENTIALS_KEY instead.)

A caveat: Claude Code currently warns against putting secrets in environment variables because there is no dedicated secrets store yet. I am using this approach because the passphrase only protects an already-restricted service account, not your personal cloud credentials. When a proper secrets store ships, this workflow will use it.

Step 2: Install the skill.

Open your repo in Claude Code and tell the agent:

"Install the cloud-bootstrap skill from https://github.com/ipeirotis/cloud-bootstrap into this repo."

(For those comfortable with a terminal, you can also run curl -sSL https://raw.githubusercontent.com/ipeirotis/cloud-bootstrap/main/install.sh | bash in any environment with access to the repo.)

Step 3: Tell the agent to set up cloud access.

"Set up GCP access for this project."

The agent will ask you for your Google Cloud project ID. Then it will look at your repo and propose a set of minimum permissions: "Based on this repo, I think the service account needs access to Cloud Storage and BigQuery. Here is why. Shall I proceed?" You approve or adjust. For a new or empty repo, it will ask what you plan to do first.

Then the agent will ask you to run a command in Cloud Shell. To open it, go to shell.cloud.google.com or click the ">_" icon in the top-right of the Cloud Console. Make sure you are in the right project, and run:

gcloud auth print-access-token

You paste the result back. This gives the agent a temporary token (valid for one hour) to do the initial setup. The agent creates the service account, grants the approved permissions, generates a key, encrypts it, commits the encrypted file, and sets up an automatic authentication hook for future sessions. The temporary token expires. From this point on, every new session starts fully authenticated. You just start working.

For teams: each person gets their own encrypted key file with their own password. The README has the details.


What to do once the ceiling is gone

Once cloud access is set up, the agent will start proactively suggesting cloud improvements when it notices opportunities: "Would it help if I moved this dataset to BigQuery so we do not have to re-process it every session?" You can also prompt this explicitly: "Can you improve your process, knowing that you have access to GCP?"

I had a dataset too large to fit in the sandbox. The agent uploaded it to BigQuery. Now I query it conversationally: "Show me the distribution of response times by condition." The agent writes the SQL, runs it, brings back the results. The data lives in the cloud permanently. No re-uploading, no re-processing.

I needed to run a survey for a research study. The agent deployed a Cloud Function with a simple web form, backed by a database. Participants visit a URL, submit responses, the data lands in a table I can query later. No server to manage. No hosting to configure. Thirty minutes from "I need a survey" to a live URL that participants were already clicking on. I still have not fully processed how absurd that is.

What does it cost? Less than you might think. Cloud Functions and BigQuery queries cost cents per run. A T4 GPU VM runs about $0.35/hour. My monthly bill for all of this is usually under $10, though a long GPU job will cost more. One practical tip: set up a billing budget alert in Google Cloud before giving the agent access. Agents can get stuck in loops, and a $10 budget alert is cheaper than finding out the hard way.


The bigger picture: finding the next loop to close

There is a trajectory here worth naming. First, the AI learned to generate: write a script, draft a document, produce code. Then it learned to execute: run the script, push the changes, create a pull request. Now it is learning autonomy: spin up a server, run the job, shut down the server, and report back. Each step closes a loop where a human used to be the connector.

The previous post gave the agent memory and a workflow. This one gives it infrastructure. Same pattern: every time you find yourself doing grunt work to connect two things that the agent should be able to connect on its own, that is a loop waiting to be closed.


What comes next

The cloud-bootstrap skill supports GCP, AWS, and Azure. It handles first-time setup, adding team members, and credential rotation (it tracks credential age and warns you after six months). It also supports multi-provider setups in the same repo and handles permission escalation gracefully: if the agent hits a permission wall, it stops and tells you exactly what role it needs and why. It never silently fails, and it never gives itself more access.

This is still early. The whole approach (encrypting credentials in a repo, pasting short-lived tokens) is a workaround, as I noted above. When proper agent identity federation arrives, this will simplify considerably. But right now it works, and for isolated research projects with tightly scoped permissions, the risk is manageable.

But the agent can still only work inside the one repo it is connected to. It cannot clone a second repo, pull in a dataset from another project, or push results somewhere a collaborator can see. It can work inside one room but cannot walk between rooms. The next post will fix that: installing gh and setting up a GitHub personal access token so the agent can move freely across repos. It is a much shorter setup than this one.

After that: the "master repo, satellite repos" setup for coordinating work across multiple projects (which needs the GitHub token to work), MCP configuration for integrating Gmail and Google Calendar, and more on the "council of LLMs" approach I have been using for grading oral exams and for reviewing my work.

But start here. Give the agent a cloud account. And then go to dinner. When you get back, the agent will have finished collecting data, training the model, shut down the GPU VM, clean up everything, and gone to sleep. Your kids, on the other hand, if they are like mine, will still be awake and making fun of the parental controls on their iPads, and the kitchen is a mess.

Wednesday, March 4, 2026

"Let's Work on the Next Task": Claude Code, GitHub, and the Most Diligent Project Manager I've Ever Had

In my previous post, I described how working with AI agents felt like managing an infinitely large, infinitely diligent team. I wrote about pairing Claude with GitHub, giving it context files and task lists, and watching it come back with actual deliverables.

After that post, I got questions from a lot of people asking how to actually set this up. Even from people I assumed were already using this kind of workflow. Turns out it was far less common knowledge than I previously thought. (I guess I am spending too much time reading social media.)

So this post is a step-by-step guide for those who still use AI tools in the "chat" form and want to examine a first setup of "agentic AI". In this case, it is not to get the AI to be a software engineer, but rather get the AI to becoming your project manager and your team of research assistants.

We will set up a GitHub repository, configure Claude Code on the Web, and build a workflow where AI plans (or does) the work and you do the reviewing.

One caveat: while you do not need to know how to code, familiarity with software development practices will help. Not the programming itself, but the process: how developers organize projects, track changes, review each other's work. This post will walk you through those practices.

First, though, let me explain why this setup is so powerful.


The real trick: The repo is the context

Here is the problem with using AI through a regular chat interface. Every time you start a new conversation, you are starting from zero. You paste in your document, re-explain what the project is about, remind the AI where you left off, describe what needs to happen next. It is like hiring a brilliant contractor who gets amnesia every morning.

GitHub solves this. When Claude Code connects to your repository, it does not just see your files. It sees everything: the project structure, the notes about what the project is, the task list, the record of what has already been done, the decisions you have made along the way... All of it, sitting right there in the repo, ready to be read.

This means your prompt for most interactions becomes absurdly simple:

"Let's work on the next most important task."

That is all. Claude reads your CLAUDE.md to understand the project. It reads your TASKS.md to figure out what needs doing. It looks at the existing files to understand the current state. And then it gets to work. No pasting. No re-explaining. No "as I mentioned in our previous conversation..." The repository is the conversation. It is the memory. It is the context.

Read about CLAUDE.md and TASKS.md and you are worried that this is some black magic? Nah, these are just regular text files, written in plain English. We will describe them next.


Wait, what is Claude Code on the Web?

First, some context. Claude Code started as a command-line tool. You would install it on your computer, open a terminal, and type commands. Powerful, but intimidating if you are not a developer.

Then Anthropic launched Claude Code on the Web. Now you can do the same thing directly from your browser. You connect a GitHub repository, give Claude a task, and it clones your repo, writes code (or documents, or reports, or whatever you need), and pushes the changes to a branch. You review the changes, approve them, and merge. All from a web interface. No installation.

Claude Code on the Web operates inside a real computing environment called the "sandbox". It can read your files, create new ones, run scripts, and push changes to GitHub. It tends to write software for performing various tasks, instead of replying in plain text. It does work. Real work. The kind you would normally delegate to a research assistant or a junior colleague.


The 10-minute setup: GitHub + Claude Code

OK, let us build this from scratch. I will assume you have zero GitHub experience.

Step 1: Create a GitHub account and a repository.

Go to github.com and sign up. Then create a new repository: click the green "New" button, give it a name (something like my-research-project or quarterly-report), make sure to set it to Private (not Public, unless you want the whole internet reading your drafts), and check "Add a README file." That last part matters. Write a short description of your project in the README. Even a couple of sentences is fine. This initializes the repo so that Claude Code can actually work with it. (An empty, uninitialized repo will cause problems.)

Step 2: Connect your repo to Claude Code.

Go to claude.ai and open Claude Code (it is in the left sidebar, or you can go directly to claude.ai/code). Start a new session and connect your GitHub repository. You can paste your repo URL directly or use the built-in GitHub integration to browse your repositories. Claude will ask you to authenticate with GitHub the first time (a one-time OAuth flow) and install Claude on the Github repo (that allows Claude to write to the repo). Select the repo you just created.

Now Claude Code can see your files, and more importantly, it can change them.

At this point, you can upload files that you have about the project to the repo, or you can defer that step for later and move on to the next step.

Step 3: Let Claude set up your project.

This is where it gets interesting. CLAUDE.md is a special file that Claude reads at the start of every session. It is the project's "master plan": what the project is about, how it is organized, what conventions to follow. But you do not need to know what it should look like. Just describe your project in plain language:

"This repo contains the data and analysis from our AI-powered oral examination system, which I wrote up as a blog post. I want to turn this into a research paper for submission to Communications of the ACM. The data and some initial analysis scripts are already in the repo. Set up the project structure for a CACM submission and create a CLAUDE.md file."

Claude will read through the existing files, figure out what is there, organize everything into a sensible structure, and create a CLAUDE.md that might look something like this:

# Project: AI-Powered Oral Examinations at Scale

## Overview
Research paper for Communications of the ACM describing our system
for conducting and grading oral examinations using conversational AI
agents and a multi-LLM grading approach.

## Submission Details
- **Journal**: Communications of the ACM
- **Format**: ACM `acmart` document class, `acmsmall` style
- **Page limit**: 12,000 words including references
- **Style**: Author-year citations (natbib)

## Structure
- `/paper/` - LaTeX source files and ACM style files
- `/data/` - Exam transcripts, grading data, survey responses
- `/analysis/` - Python scripts for statistical analysis
- `/figures/` - Generated plots (PDF format, generated from scripts)
- `/blog/` - Original blog post and supporting materials

## Conventions
- All figures must be generated from scripts in `/analysis/`,
  never created manually
- Use BibTeX for references (`references.bib`)
- Data files are never edited directly; all transformations
  happen through scripts in `/analysis/`
- Student data must be anonymized in all outputs

## Current Status
See TASKS.md for the current task list and priorities.

Notice: you did not write any of this. You described your project, and Claude produced the project master plan. You review it, maybe tweak a couple of things. Done.

Step 4: Create your TASKS.md file.

This is your project's to-do list. But unlike a regular to-do list, it serves double duty: it tells Claude what needs to be done and keeps a record of what has been completed. Ask Claude to create it:

"Create a TASKS.md file with the following initial tasks..."

Here is what one might look like:

# Tasks

## In Progress
- [ ] E1. Expand blog analysis into formal experimental evaluation
- [ ] E2. Inter-rater reliability analysis (human vs. LLM council grades)

## To Do
- [ ] E3. Create Figure 1 (grade distribution across grading methods)
- [ ] R1. Write Related Work section (AI in assessment, LLM-as-judge)
- [ ] D2. Analyze anti-cheating detection rates
- [ ] Z3. Check word count against CACM 12,000-word limit

## Done
- [x] Z1. Set up project structure from blog post materials
- [x] D1. Anonymize student data
- [x] I1. Write Introduction draft

Now here is the magic. You can point Claude at a specific task and say: "Work on the next task in TASKS.md." Claude reads the file, picks the next item, does the work, updates the task status, and creates a pull request with its changes. If you are not familiar with pull requests, more in a moment.

Step 5: Give Claude a GitHub token (so you never have to learn git).

There is one more thing worth setting up now, even though it will not feel essential until later. Claude Code on the Web can push changes and create pull requests through its built-in GitHub connection. But that connection is limited to the one repo you connected in Step 2. If you want Claude to handle all the git operations fluently, and eventually work across multiple repos, you need to give it a personal access token.

Go to github.com/settings/tokens, click "Generate new token (classic)," give it a name like claude-code, and select the repo scope (which covers reading, writing, and managing repositories). Copy the token.

Now go back to Claude Code, open the environment settings for your session, and add:

GITHUB_TOKEN=ghp_your_token_here

Then ask Claude to add this line in the CLAUDE.md:

## Global Tools
- `gh` (GitHub CLI) is available as a global tool, authenticated via the `GITHUB_TOKEN` environment variable.

That is it. Next time a session starts, Claude will install gh (the GitHub command-line tool) and authenticate using your token. From that point on, Claude handles all the git plumbing: committing, branching, creating pull requests, even cloning other repos when needed. You never run a git command. You never resolve a merge conflict. You click "Merge" on the pull request, and Claude takes care of everything else.

Why does this matter? Because without it, you will eventually hit a moment where Claude says something like "I've made the changes but I need you to run git pull and resolve a conflict." And suddenly you are googling git tutorials at 11pm, which is not the workflow we are going for.

(This token also unlocks cross-repo operations, which becomes important once you start coordinating work across multiple projects. More on that in a future post about the "master repo, satellite repos" setup.)


Pull requests: Redlined documents for coders (and not only)

Now the part that is unfamiliar to people who are not software engineers. The "pull request".

If you have ever received a redlined document from a lawyer, or reviewed tracked changes in a Word file, you already understand pull requests. The concept is that simple: someone proposes changes, you review them before they get incorporated into the main document.

In GitHub, it works like this:

  1. Claude does its work on a separate branch (a parallel copy of your project).
  2. When it is done, it creates a pull request (PR), which says: "Here are the changes I made. Want to incorporate them?"
  3. You see a clean diff view showing exactly what was added, removed, or modified. Green lines are additions. Red lines are deletions.
  4. You review. You can approve, request modifications, or reject.
  5. If you approve, you click "Merge" and the changes become part of the main project.

This is the standard process used by every software team in the world. And it works for any kind of knowledge work that relies on text. Research papers. Reports. Course materials. Business proposals. Anything that lives in files. Ideally, you want the files to be text files and not binary ones; tex good, PowerPoint files, not so much. In the future we may have better tooling for reviewing changes in Office files or other formats, but for now the process works best for text-based files.

Fair warning: the GitHub interface will look busy the first time you open a pull request. Do not panic. Just look for the "Files changed" tab to see the redlines, and the big green "Merge pull request" button when you are ready to accept.

The critical point: you never edit the files directly. You describe what you want, Claude proposes changes, and you review and approve. You are the manager. Claude is the diligent employee who comes back with deliverables for you to inspect. And the audit trail is far better than "Track Changes" in Word ever was.


A real example: From CSV to submission-ready in two hours

Let me show you how this plays out in practice with a real example from last month.

I was working on a paper that had a case study section (say, Section 8) where we discussed results from a partner's dataset, but we only had the final business conclusions, not a full experimental analysis. The rest of the paper (say, Section 7) had a proper, thorough analysis on a different dataset: figures, tables, bootstrap confidence intervals, the works. By comparison, the case study in Section 8 was the weak sibling, and reviewers have flagged that. We have received a detailed dataset from our partners, but it required work. My TASKS.md had this sitting in it:

## Backlog
- [ ] F5. AML dataset analysis
- [ ] G1. Complete §8 rewrite with AML dataset

I uploaded the CSV to the repo and told Claude:

"Here is the AML dataset. Replicate the analysis from Section 7 but now for Section 8. Use the existing details from Section 8 as the background and framing, conduct the full experimental analysis, and generate a new Section 8."

Claude read Section 7 to understand the methodology. It read the existing Section 8 to understand the framing and context. It wrote Python scripts to process the AML data, generated four figures and three tables with bootstrap confidence intervals, wrote the new section text with all quantities pulled from the analysis scripts, and submitted a pull request with everything.

Less than an hour. I spent another hour reviewing the PR, checking the code, leaving comments ("clarify this axis label," "move this paragraph before the table", "I do not think the conclusions follow from the results"), and merging.

Two hours total. For a PhD student, this would have been a few days of work, easily. And here is the part that matters: every single number in that section was generated through a Python script. Every figure had a script that produced it. Reproducibility was built in from the start, not bolted on after the fact. The pull request showed me exactly what was added: the scripts, the outputs, the LaTeX changes. I could trace every claim back to the code that produced it.

Needless to say, I remain fully accountable for any bugs or errors. At the end of the day, I have reviewed the scripts, the results, and the text. What I can say is that even if there are errors, these are not "hallucinations" where the LLM filled in random numbers or references in the text. The figures are Python-generated from the raw data, the tables and the numbers in the text the same. The errors can come from bugs, or other oversights. But we should stop calling all AI errors "hallucinations". At this point, the errors are not the errors of a "bullshitter in chief" (a title aptly earned by early LLMs); they are the same types of errors that a junior colleague may make when carefully executing a well-defined task: misreading a specification, applying a method slightly outside its intended scope, or missing an edge case that a more seasoned eye would have caught.


Beyond software: Why this works for all knowledge work

I want to be explicit about something: this is not just for code. GitHub repositories can hold any kind of file. Markdown documents, LaTeX papers, CSV data files, images, PDFs. The pull request workflow works for anything.

Writing a consulting report? Put the markdown draft in /report/, the supporting analysis in /data/, the charts in /figures/. Claude generates the analysis, creates the figures, and drafts sections of the report, all as reviewable pull requests.

Same idea for course materials (I use this with my exit tickets workflow), business plans, grant proposals. You define the project structure, you maintain a task list, and you let the agent do the work while you review proposals. Standard software engineering practice, applied to everything.


Leveling up: More files for better project management

Once you get comfortable with CLAUDE.md and TASKS.md, you can add more structure. The files I have found most universally useful are these three:

  • SCHEDULE.md — Deadlines and milestones. "The submission deadline is March 15" becomes a constraint that shapes which tasks get prioritized first.
  • DECISIONS.md — Key choices and their rationale. "We decided to use three LLMs in the grading council instead of five because the marginal improvement was negligible." Prevents you and Claude from relitigating settled questions two weeks later.
  • STYLEGUIDE.md — Your writing preferences. "Never use em-dashes," "Never use fluffy adjectives," "Avoid claims not supported by data or citations." Good trick: give Claude a few pieces of your favorite writing and ask it to generate a style guide that mimics your voice. Then drop it in the repo.

Beyond these, there are files worth adding for specific situations:

  • CHANGELOG.md — Human-readable log of what changed each session. Especially useful when preparing a response to reviewers.
  • BLOCKERS.md — Things waiting on someone external. Makes it easy to send a collaborator a list of "here is what I need from you."
  • FEEDBACK.md — Running log of all feedback received, formal and informal, with status: pending, accepted, or rejected with rationale.
  • SOURCES.md — Annotated bibliography: what each source is useful for, how reliable it is, which sections cite it.
  • GLOSSARY.md — Keeps terminology consistent across a long document. Claude consults it and adds new terms as they come up.
  • DEPENDENCIES.md — Maps how artifacts depend on each other. Lets Claude flag when an upstream change invalidates something downstream.

You do not need all of these on day one. Start with CLAUDE.md and TASKS.md. Add CHANGELOG.md when editing a paper that came back with revisions. Add the rest as your project grows and you find yourself needing them.

To be fair, this is a bit of a hack. We are simulating standard project management tools using plain markdown files. Scanning text files for task lists and decisions is not exactly elegant. And I have serious doubts that this can scale for projects involving hundreds of people. But it works for now, with tools that exist today, for the projects that I am working on.

In the future, agents will have proper interfaces: structured databases, purpose-built PM tools designed for agents to read and write directly, not markdown files they have to parse every session. We are in the duct-tape-and-baling-wire phase. It is fine. The duct tape holds.


The awkward part (and why it is worth it)

If you are not a software engineer, this workflow feels strange at first. You are used to opening a document and typing. Now you are writing instructions, waiting for an AI to propose changes, and clicking "Merge" on a pull request. It is indirect. It feels like you are adding a middleman.

But here is what happens after a week: you realize the middleman can do 80% of the work. And the 20% you are doing (reviewing, giving feedback, making decisions) is the work that you would have done with any apprentice. But you are not fixing typos, you are not formatting tables, you are not wrestling with matplotlib's axis labels. You are reading the output and deciding if it is good and trustworthy enough.


Coming next

This post covered the basics: one repo, one project, Claude Code on the Web doing the work. The whole secret is that now the chatbots can write down what they have done, and look up the notes next time you start working together. And it is ridiculously powerful.

But this is just the beginning.

In upcoming posts, I will describe my "master repo, satellite repos" setup, where I maintain a central task management repository that coordinates work across multiple projects with different collaborators. Think of it as the command center.

Beyond that: deploying resources on Google Cloud, spinning up virtual machines for heavy computation, and the "council of LLMs" approach where Claude, Gemini, and GPT deliberate together on evaluation tasks (something I have been using for grading oral exams and am now extending to research).

At some point (in the not so distant future, probably by the end of March or so) Claude will be scheduling my meetings, answering my emails, and assigning me tasks from my own task list. I am not entirely sure who is managing whom anymore.

Sunday, February 15, 2026

Listening to My Students at Scale: Exit Tickets, NotebookLM, and the Tightest Feedback Loop I've Ever Built

It started at a teaching workshop, last semester: Craig Kapp and Rob Egan presented a seminar at the NYU Center for Teaching and Learning called "Real-Time Insights: Leveraging AI for Responsive Teaching in Large Classrooms." They (re-)introduced a deceptively simple concept: the exit ticket. The idea is that at the end of every class session, you ask students three quick questions, each with a different shape metaphor:

  • 🔵 Circle: What is still circling in your mind? (What are you confused about?)
  • 🟥 Square: What "squared" with your understanding? (What clicked today?)
  • 🔺 Triangle: What are three key takeaways from today's session?

Then, take these answers, and use LLMs to process them quickly and get feedback before the next session.

Getting structured feedback from students after every single session? Not at the end of the semester when it's too late to change anything, but right now, while you can still do something about it? I immediately wanted to try it.

Below I describe the details of the approach presented by Craig and Rob, and my own adjustments to the recipe. Hope you will find it useful.


The setup: Making it required (and why that matters)

It starts by setting up the exit ticket surveys as auto-graded quizzes on Brightspace (NYU's LMS). The auto-grading part is a nice little trick: one of the questions is simply "Select True in this question to get your points." Students complete the survey, they get their credit. No manual processing of ~50 submissions on my end.

We do tell students upfront: write something substantive. Don't game the system. We reserve the right to deduct points if someone slacks through the exit tickets all semester. And here's the nice irony: since we're already running AI-powered analysis on the responses, identifying freeriders who type "asdf" every week is trivial. The same pipeline that processes the feedback also flags the people not providing any.

The critical design decision: make it part of the grade, not optional. Optional feedback gets ~30% response rates and self-selected complainers. Required feedback gets everyone. And because this is formative feedback (not evaluative), students have every reason to be honest and detailed. They're not rating me. They're telling me what they need.

Compare this to the end-of-semester evaluation. Students fill it out in December, the professor reads it in January (maybe), and any changes happen next year for a completely different group of students. The feedback loop is so long that it barely qualifies as a loop. Exit tickets close that loop within days. Sometimes hours.


From exit ticket to next session: the processing pipeline

So now I have all this feedback. ~50 students, after every session, telling me what confused them, what clicked, and what they're taking away. The question becomes: how do you actually process all of that quickly enough to act on it?

NYU IT built an official path for this, which Rob demonstrated in the seminar. You export the exit ticket responses into the Brightspace Insights Portal (which Rob's team manages) and run AI-powered analysis using a prompt like this:

You are an expert Instructional Designer and Data Scientist assisting
a professor with the course "AI/ML Product Management" at NYU Stern
School of Business (undergraduate).

Your goal is to analyze student feedback survey data to improve course
delivery. The survey questions and student answers are provided below.
Please perform the following two steps:

### Step 1: Thematic Analysis
Analyze the responses to identify key themes. Do not just look for
keywords; look for semantic similarities and underlying sentiment. For
each theme, provide:
1. **Theme Name**: A concise title.
2. **Prevalence**: The approximate number of students who mentioned this.
3. **Explanation**: A brief summary of the sentiment or issue.
4. **Evidence**: A direct, representative quote from the data.

### Step 2: Actionable Pedagogy (Bloom's Taxonomy)
For each theme identified above, propose a short course activity.
* If the theme represents a **knowledge gap/pain point**, propose a
  remedial activity.
* If the theme represents a **strength/interest**, propose an activity
  to deepen understanding.
* **Constraint**: The activity must be supported by Bloom's Taxonomy.
  Explicitly state which level of Bloom's Taxonomy the activity targets
  (e.g., Application, Analysis, Evaluation).

**Format**:
Start the suggestion section for each theme with the label: "PRACTICE IDEA".

I attach the survey data.

It's a well-designed prompt. Thematic coding, prevalence counts, representative quotes, remedial activities aligned with Bloom's Taxonomy. The output is genuinely useful.

But I prefer to do something slightly different. I use the same prompt from the Insights Portal, but I run it inside NotebookLM with just the student feedback as input. For those unfamiliar: NotebookLM is Google's AI-powered research assistant. You upload your own documents, and it generates analysis, summaries, explainer videos, and podcast-style audio overviews grounded entirely in your uploaded sources. NYU provides institutional access through Google Workspace, so the data never trains any AI models, which matters when you're working with student feedback.

Why NotebookLM over the Insights Portal? Because the exit ticket analysis is just the starting point. What I really need is to prepare the follow-up material. Once NotebookLM identifies the themes and suggests activities, I take those suggestions and combine them with my lecture slides, readings, and case studies (which are already loaded in the same notebook). Then I ask it to generate explainers, videos, infographics, and targeted activities that address the confusion, all grounded in my actual course content.

The Insights Portal gives me a diagnosis. NotebookLM gives me the diagnosis and helps me build the treatment.

My workflow after every class:

  1. Students complete the exit ticket on Brightspace (takes them 2-3 minutes)
  2. I export the responses and upload them into a NotebookLM notebook, together with the materials for that session
  3. NotebookLM identifies the themes: what's confusing people, what clicked, what they found most valuable
  4. Based on those themes, I generate explainer materials, short videos, and targeted activities for the next session

(As an example, here is the NotebookLM that we use for the Zillow Offers case, which we use to discuss leading and lagging metrics, model and output monitoring, concept drift, adverse selection and other product-management-related topics. Note: this notebook contains only course materials for preparing the case discussion, not student feedback data.)

One small but annoying wrinkle: NotebookLM's default slide output has that unmistakable "AI-generated" aesthetic. You know the one. (Yes, they are visually gorgeous compared to my own slides, but after a while it starts feeling a bit like slop.) So I started uploading the NYU brand style guide as an additional source in my notebooks, and prompting NotebookLM to follow it when generating visual materials. The results are noticeably closer to proper NYU-branded slides. Not perfect, but much better than the generic AI look. I'm still waiting for NotebookLM to support custom templates or branding natively, but that's a different story.

The per-session overhead is maybe 15-20 minutes.


Why this actually works

The circle/square/triangle structure does something clever: it gives students permission to be confused. "What is still circling in your mind?" is a much less intimidating question than "What don't you understand?" And the three-takeaways question forces them to reflect, even briefly, which helps consolidate their learning.

But the real reason students engage is that they see the results. When I open the next class by saying "Several of you mentioned you were confused about X, so let's spend 15 minutes on this before we move on," students learn that their feedback actually matters. It creates a virtuous cycle: they write thoughtful responses because they know I'll respond, and I can respond because NotebookLM makes processing all the responses feasible. Without the AI assist, no professor has time to synthesize free-text responses from 50 students after every class and create targeted follow-up materials. Definitely not after every single session. The economics just don't work.

With NotebookLM doing the heavy lifting? The economics suddenly work beautifully.

The exit ticket has been around for decades. Craig and Rob simply showed how to supercharge it with AI. The hard part was never getting students to talk. It was finding the time to listen. Once students realize someone is actually listening, they start saying things worth hearing. That's the loop. That's the whole trick.