Bidding Information Crawler for a Friend, But I Started Wrong from the Beginning

image.png

Recently, I helped a friend create an automated tool for gathering bidding leads. His requirement was actually quite simple: based on the company's business scope, search for relevant bidding information in search engines daily and filter out potentially valuable leads.

He had been doing this manually, spending a significant amount of time each day. So, he hoped for an automated tool to replace this repetitive labor.

Initially, I also thought this was a very simple requirement. Web scraping technology is quite mature, and using an AI IDE should be enough to write the project. Moreover, since we were already using AI, why not add a bit of LLM capability, such as automatically parsing company materials to generate keywords, and incorporating LLM evaluation and analysis into the process.

So, I quickly got to work.

First, I discussed the requirements with Gemini, wrote a requirements document, and created an interaction specification. Then, I placed both documents in the project root directory, letting Codex automatically generate a development plan based on the documents and implement it step by step. Since this tool is a standalone application using Playwright for browser automation, not a large-scale web scraper, it was relatively friendly in terms of implementation difficulty and compliance.

After a couple of days of whack-a-mole debugging, I finally produced the first version. Users could upload company materials, the LLM would parse the information and generate keywords, and then call Playwright to execute searches in search engines. Users could also upload company qualifications, and the system would use the LLM to evaluate and match each piece of bidding information captured.

But the results were not ideal.

Many portal websites' results couldn't be drilled down further, only capturing a title. The result list was mixed with a large amount of news content, and some bidding information was even long expired. After repeated discussions with the AI, I tried strengthening the filtering rules and switching from portal website list pages to targeted scraping, but the effect was still mediocre.

Staring at the hideously ugly interface again, I quickly realized where the problem lay.

image.png

I asked the AI to help me implement a tool, but what it produced was a tool from a bygone era.

Although I had never done web scraping before, I also knew that writing rules, cleaning data, and maintaining site adaptations were inherently complex engineering tasks. And I already had AI, yet I was still solving problems in the traditional way.

So, my current problem wasn't technical; it was in my personal way of thinking.

My product mindset was still stuck in the past, only seeing AI as a "faster programmer." But AI isn't just for writing tools; it is itself part of the tool.

Therefore, in the second stage, I began adjusting the direction, reducing rules and structured processes, and shifting towards semantic processing.

While solutions like browser-use are more intelligent, their token consumption is too high. So, I chose to add a simple layer of intelligent decision-making on top of Playwright, giving the program some understanding capability during browsing and clicking.

At the same time, I also rethought the project's true goal. Its goal wasn't to be "better," but to "replace." It only needed to replace my friend's daily manual search process, freeing him from spending time on repetitive operations and allowing him to focus on more critical tasks. As long as it achieved this, it would already save him 90% of the time he spent on this task.

Looking back, many things I did earlier were actually superfluous.

For example, more complex keyword generation and more detailed data evaluation. This work didn't need to be done for him by AI, because in the past, he used his own keywords to search and could quickly judge which information was valuable, indicating he already possessed this capability. What truly needed to be replaced was only the search itself.**

Overdeveloping these analytical capabilities actually led me astray from the core problem.

Humans are always the subject of the matter; tools are merely extensions of humans.

This incident also made me rethink a larger issue.

Some scenarios indeed require AI to participate in value judgments, such as when I use AI to write code, because many technical problems are beyond my capabilities. But most of the time, people don't need to completely outsource their value judgment ability to AI.

The tool for my friend is still under development, but this story of making mistakes can end here.

Just like many companies implementing intelligent customer service, it's actually to reduce costs, or even to block responsibility. But the original purpose of the customer service role was to soothe consumer emotions, collect genuine feedback, and thereby improve products and service experiences. It was itself a channel connecting enterprises and consumers, only later did many companies turn it into a "firewall."

Now, with AI, it's actually an opportunity for re-division of labor. Repetitive, mechanical, and procedural communication can be handed over to AI, while human customer service can focus more energy on understanding consumers, solving problems, and optimizing products. Let professions return to value rationality.

Technology should not diminish human value. What it truly needs to take over are those dead-end processes that make people more mechanical.