Show HN: Simplex: Automate browser workflows using code and natural language

Yeah sure! We mentioned a little bit in the post that we stumbled upon this problem when working on a synthetic data contract. Internally, we were building software at the time to fully automate the process of creating synthetic data, all the way from purchasing assets to building scene layouts to rendering images. We realized after 2 months of building that there wasn't a great need for this problem nor could we build something minimal but feature-complete that we could give to people and iterate off of. We also saw the writing on the wall re: building 3D world models (see WorldLabs and NVIDIA's new 3D world models). We're excited to see where this goes!

VC funding.

I thought it's related to the simplex method in linear programming

I for one did not make that connection. Only thing I thought of was https://simplex.chat/ (which is fortunately not marketed as a dating app)

RE: feedback mechanism -- yep, a feedback mechanism is definitely something we're thinking of adding. Since we use VLMs that are trained to always output coordinates (i.e., they don't have a way to say "not on the page"), we're probably going to try fine tuning with some negative examples to see if we can build that feedback mechanism in.

One way to hack the scrolling to an element is to first run extract_bbox on a natural language description (in your case for GitHub it might be "follow button") then take the Y coordinate of that element and scroll that number of pixels. I just wrote this bit of code that I tested and it brings the contribution graph into full view:

    simplex.goto("github.com/spro")
    coords = simplex.extract_bbox("follow button")
    simplex.scroll(coords[1])
    simplex.wait(2000)
    image = simplex.extract_image("green tile graph")

But then it incorrectly picks the code review/submissions/etc. graph as the green tile graph -- we'll look into it!

re: frequency graph typo -- just pushed a fix, thanks!

So sorry about this, it looks like our servers went down overnight. Just fixed, should be good now!

Thanks John! We should make that more clear in the docs. You can set browser=None on initialization and Simplex will create a local browser instance that can run on your local websites. We're not planning to be open source right now since a large portion of our product is custom vision models + inference speedups through hosting.

Browser Use is another YC company. Probably the biggest difference is that they're more agent focused while we're more lower level -- in the Claude Computer Use camp like you mentioned.

Hey thanks! We think we're pretty different from Skyvern as Skyvern provides a full agent loop for users + is no code. We wanted to build something at a lower level (i.e., no high-level planner that chooses what tasks to do for you) and we wanted to be able to program our own logic alongside the intelligence part because using code is what's natural for us as developers.

I hadn't heard of UI Vision but just took a look at it -- it also looks like a no-code solution that's a Chrome extension, so I'd say the main differences are the same as the differences w/ Skyvern -- we're lower level and meant to be used by developers.

I'd add that we're also able to directly extract parts of websites that have no official API -- for example, an image of the GitHub contribution graph like I show in the video demo.

We finetune our own VLMs for this -- unfortunately prefer not to share which ones we use specifically! ClickClickClick looks awesome, have you heard of FerretUI (https://arxiv.org/pdf/2404.05719)? Pretty similar idea.

I'm surprised you named your framework clickclickclick instead of taptaptap.

extract_text returns all the lines of text it finds within an element. Looks like for your case it selected a majority of the page from the description, so it's returning most of the text on the page.

We currently don't return failure cases (just closest match) -- but good suggestion! We'll fine tune on some negative cases and see if we can catch them

Thanks! We're happy to add more REST API functionality -- could you email us at founders@simplex.sh if you're up to talk further about what you'd like to see? RE: pricing, we're still figuring this out!

Thanks, we hope so! That's an interesting conversation -- I'd argue that the graph-based no-code solutions you're referring to are for a different set of people than those commonly found on HN. As a developer I didn't particularly want to work with those tools, so we built this to supercharge our own code-based flows instead. I actually don't think the node-based flows are horrible at all since they successfully enable non-developers/less technical people to use agents/build easy automations.

Yep, any websocket URL works. I see that Browserless offers a websocket URL, so you can use them! You just need to pass in a Playwright browser object into the Simplex constructor.

Ah, we've whitelisted some websites to prevent abuse -- amazon.in wasn't one of them. I just whitelisted amazon.in for you and tried the search query -- looks like it works!