205 points | by mohsen11 month ago
The way I currently do this is that I wrote a small python file that I can start with
llmcode.py /path/to/repo
It then creates a prompt like this:
Look at the code files below and do the following:
{task_description}
Output all files that you need to change in full again,
including your changes. In the same format as I provide
the files below, that means each file starts with
filename: and ends with :filename
Under no circumstances output any other text, no additional
infos, no code formatting chars. Only the code in the
given format.
Here are the files:
somefile.py:
...code of somefile.py...
:somefile.py
someotherfile.py:
...code of someotherfile.py...
:someotherfile.py
assets/css/somestyles.css:
...code of somestyles.css...
:assets/css/somestyles.css
etc
I then look at the changes via "git diff".
It's quite fascinating. I often only make minor changes before accepting the "pull request" the llm made. Sometimes I have to make no changes at all.
This might sound silly, but I feel like it has the potential of resulting in more readable code.
There have been times where I split up a 300 line function just so it’s easier to feed into an LLM. Same for extracting things into smaller files and classes that individually do more limited things, so they’re easier to change.
There have been times where I pay attention to the grouping of code blocks more or even leave a few comments along the way explaining the intent so LLM autocomplete would work better.
I also pay more attention to naming (which does sometimes end up more Java-like but is clear, even if verbose) and try to make the code simple enough to generate tests with less manual input.
Somehow when you understand the code yourself and so can your colleagues (for the most part) a lot of people won’t care that much. But when the AI tools stumble and actually start slowing you down instead of speeding you up and the readability of your code results in a more positive experience (subjectively) then suddenly it’s a no brainer.
But now, to actually improve my own productivity a lot? I’ll dig in more often, even in messy legacy code. Of course, if some convoluted LoginView breaks due to refactoring gone wrong, that is still my responsibility.
At work this week I was investigating how we could auto scale our CI. I know enough Jenkins, AWS, perforce, power shell, packer, terraform, c++ to be able to do this, but having the time to implement and flesh everything out is a struggle. I asked Claude to create an AMI with our workspace preloaded on it, and a user data script that set up a perforce workspace without syncing it, all on windows, with the tools I mentioned. I had to make some small edits to the code to get it to match what I wanted but for the most part it took 2-3 days screwing around with a pretty clear concept in my head, and I had a prototype running in 30 minutes. Turns out it’s quicker to sync from perforce than it is to boot the custom AMI , but I learned that with an hour in total rather than building out more and we got to look at alternatives. That’s the future to me.
> edit a lot of lines
"a lot" being the keywords here.
I am personally torn between the future of LLMs in this regard. Right now, even with Copilot, the benefit they give fundamentally depends on the coder that directs them - as you have noted.
What if that's no longer true in a couple years? How would that even be different from e.g. no code tools or website builders today? In different words will handwritten code stay valuable?
I personally enjoy coding so I can always keep doing it for entertainment, even if I am vastly surpassed by the machine eventually.
> I personally enjoy coding so I can always keep doing it for entertainment, even if I am vastly surpassed by the machine eventually.
I agree with both these takes, and I think they’re far more important than wondering if hand written code is valuable.
I do some DIY around the house. I can make a moderately straight cut (within tolerances for joinery use cases). A jig or circular saw makes that skill moot, but knowing I need a straight clean cut is a transferable skill. There’s also two separate skills - being able to break down and understand the problem and being able to implement the details of the problem. In trade skills we don’t expect any one person to design, analyze, estimate, build, install and decorate anything larger than a small piece of furniture and I think the same can be said of programming.
It’s similar to using libraries/framesorks - there will always be people who will write shitty apps with shitty unmaintainable code - we’ve been complaining about that since I’ve been programming. Those people are going to move on from not understanding their wix websites to not understanding their AI generated code. But it’s another tool in the belt of a professional programmer
> "I wrote a small python file that I can start with"
Which one is it, chief?
They mean the same thing, chief.
Serializing an entire repository and feeding it to an LLM, and having the LLM change the code in response to specific asks, then storing the results - is not a trivial thing to implement. Think about large repositories with very long files, or repositories with a complex structure, or polyglot repositories. Projects where code and data are mixed together. Then think about security, with things like prompt injection in the source files, etc. Handling hallucinations.
I like to be a "builder of systems" , "solver of problems". "Organizer or manager" would also fit in that description. And then what tool you use to get stuff done is not relevant.
> You will organize the codebase in a way AI can handle it, make architectural decisions and organize the workflow around AI doing the actual coding.
https://prompt.16x.engineer/cli-tools
I also built a GUI tool that does this:
Here is a benchmark comparing it to [Repomix][1] serializing the Next.js project:
time yek
Executed in 5.19 secs fish external
usr time 2.85 secs 54.00 micros 2.85 secs
sys time 6.31 secs 629.00 micros 6.31 secs
time repomix
Executed in 22.24 mins fish external
usr time 21.99 mins 0.18 millis 21.99 mins
sys time 0.23 mins 1.72 millis 0.23 mins
[chimeracat](https://github.com/scottvr/chimeracat)
It took the shape that it has because it started as a tool to concatenate a library i had been working on into a single ipynb file so that I didn’t need to install the library on the remote colab, thus the dependency graph was born (as was the ascii graph plotter ‘phart’ that it uses) and then as I realized this could be useful to share code with an LLM, started adding the summarization capabilities, and in some sort of meta-recursive-irony, worked with Claude to do so. :-)
I’ve put a collection of ancillary tools I use to aid in the pairing with LLM process up at https://github.com/scottvr/LLMental
Of course, if you meant “hilarious” similarly to how I mean “ludicrous”, thanks! And thank you for taking the time to look at it. :-)
Your project is amazing, you are ahead of me in your thinking.
If you haven't check out
Synthesizing Bijective Lenses https://arxiv.org/abs/1710.03248
Synthesizing Symmetric Lenses https://arxiv.org/abs/1810.11527
Synthesizing Quotient Lenses https://www.cs.sfu.ca/~miltner/papers/qptician.pdf
https://github.com/Optician-Tool/Optician-Tool
Boomerang https://www.seas.upenn.edu/~harmony/
https://www.microsoft.com/en-us/research/group/prose/
The subject touches on program synthesis, type directed programming and programming with holes.
Thank you for the rest of the links!
Thanks again.
It outputs both a file tree of your repo, a list of the dependancies, and a select list of files you want to include in your prompt for the LLM, in a single xml file. The first time you run it, it generates a .project-context.toml config file in your repo with all your files commented out, and you can just uncomment the ones you want written in full in the context file. I've found this helps when iterating on a specific part of the codebase - while keeping the full filetree give the LLM the broader context; I always ask the LLM to request more files if needed, as it can see the full list.
The files are not sorted by priority in the output though, curious what the impact would be / how much room for manual config to leave (might want to order differently depending on the objective of the prompt).
The idea would be to make it more token efficient and (lower accidental perplexity), e.g. by renaming variable names, fixing typos and shortening comments.
It should probably run after a normal linter like black.
I have to remember this.
All these other ways seem unnecessarily complicated...
Can you share your function, please?
(I’ve been using RepoPrompt for this sort of thing lately.)
def get_important_files(self, file_tree):
# file_tree = "api/backend/main.py api.py"
# Send the prompt to Azure OpenAI for processing
response = openai.beta.chat.completions.parse(
model=self.model_name,
messages=[
{"role": "system", "content": "Can you give the list of upto 10 most important file paths in this file tree to understand code architechture and high level decisions and overall what the repository is about to include in the podcast i am creating, as a list, do not write any unknown file paths not listed below"}, # Initial system prompt
{"role": "user", "content": file_tree}
],
response_format=FileListFormat,
)
try:
response = response.choices[0].message.parsed
print(type(response), " resp ")
return response.file_list
except Exception as e:
print("Error processing file tree:", e)
return []
Removed & tried again this was the result. Is the SHA256 mismatch a security concern?
Edit: Working on a fix here https://github.com/bodo-run/yek/pull/14
You can use the bash installer on macOS for now. You can read the installer file before executing it if you're not sure if it is safe
The priorization mentioned in the readme is especially interesting.
https://github.com/bodo-run/yek/blob/17fe37fbd461a8194ff612e...
I think "the part of it" is key here. For packaging a codebase, I'll select a collection of files using rg/fzf and then concatenate them into a markdown document, # headers for paths ```filetype <data>``` for the contents.
The selection of the files is key to let the LLM focus on what is important for the immediate task. I'll also give it the full file list and have the LLM request files as needed.
Is it purpose-built for code, or any text (e.g., Obsidian vault) would work?
One thing I have with llmctx that I think is missing in yek is a “Claude mode”, as it outputs the concatenation in a format more suitable to provide context for the Anthropic LLM.
Note, I understand why code context is important for LLMs. I don't understand what this chunking is or how it helps me get better code context.
Chunking is useful because in chat mode you can feed more than context max size if you feed in multiple USER messages
LLMs pay more attention to the last part of conversation/message. This is why sorting is very important. Your last sentence in a very long prompt is much more important the first.
Use case: I use this to run an "AI Loop" with Deepseek to fix bugs or implement features. The loop steers the LLM by not letting it go stray in various rabbit holes. Every prompt reiterates what the objective is. By loop I mean: Serialize repo, run test, feed test failure and repo to LLM, get a diff, apply the diff and repeat until the objective is achieved.
> in chat mode you can feed more than context max size if you feed in multiple USER messages
Just so you know, this is false. You might be using a system that automatically deletes or summarizes older messages, which would make you feel like that, and would also indicate why you feel that the sorting is so important (It is important! But possibly not critically important).
For future work, you might be interested in seeing how tools like Aider do their "repo serializing" (they call it a repomap), which tries to be more intelligent by only including "important lines" (like function definitions but not bodies).
it has some nice features:
- follows .gitignore patterns
- optional file watcher to regenerate the snapshot when files are changed
- supports glob patterns for filter/exclude
- can report token usage
- can use a config yaml file if you have complicated filter/exclude patterns
i find myself using it all the time now. it's great with claude projects
Anyone has a bash script that covers this use case?