preloader
post-thumb

Last Update: March 27, 2025 7 min read


BYauthor-thumberic

|Loading...

tyo-crawler: Crawl, Clone and Log In to Websites with Puppeteer and Redis

Short version: tyo-crawler is a Node.js command-line crawler. It drives a real Chrome through Puppeteer, so it can read pages that only render with JavaScript and pages behind a login. It keeps its link queue in Redis, so a crawl can be stopped and picked up again. You run it like this:

bash
git clone https://github.com/tyolab/tyo-crawler.git
cd tyo-crawler
npm install
# Redis must be running on localhost:6379 (see below)
node index.js --clone true https://www.example.com/

If the site is plain static HTML, you don't need a browser at all: wget --mirror is simpler and faster (see "When to use something else" near the end).

I wrote tyo-crawler. An older version of it, built on Nightmare.js, has been in my repo since 2023. In March 2025 I rebuilt it on Puppeteer: the browser was launching on 26 March, logging in to a page worked the same day, and cloning a website worked on 27 March. Since then I've used it with custom processors to collect tweets from X and to turn shared Evernote notes into Markdown. This post covers the crawler itself. Everything below is checked against the code in the repo as of its latest commit (December 2025). The README is behind the code in a few places, and I point those out.

What you need

  • Node.js 18 or later. The repo uses Puppeteer 24, and npm install downloads a matching Chrome build into ~/.cache/puppeteer.
  • Redis on localhost:6379. The crawler connects on start-up, every time.
  • Optionally curl, if you use --with-curl.

Starting Redis

Either install it from your package manager:

bash
sudo apt-get install redis-server
sudo systemctl start redis-server
redis-cli ping   # should print PONG

or use the Compose file in the repo's docker/ folder:

bash
cd docker && docker compose up -d && cd ..

That Compose file publishes port 6379 on all interfaces with no password (ALLOW_EMPTY_PASSWORD=yes). On a laptop behind a router that's usually fine. On a server with a public IP, anyone can then read and change your Redis, including the login cookies the crawler stores there. Bind it to localhost instead:

yaml
    ports:
      - "127.0.0.1:6379:6379"

If the bitnami/redis image won't pull for you, the official redis image works the same way for this purpose (image: redis:7, and mount the data folder at /data instead of /bitnami/redis/data).

The three ways to run it

What the crawler does depends mainly on two switches, --seed and --clone:

Mode
How to start it
What it does
Follows links?
Single page
node index.js URL
Opens one page in the browser, runs any actions, saves the rendered HTML
No. Links it finds are added to the Redis queue but not visited
Site crawl
node index.js --seed true URL [URL...]
Visits the seed pages, then every queued link on the same host(s)
Yes, <a> links only, on the seed hosts (plus --pattern)
Clone
node index.js --clone true URL
Like a site crawl, but also downloads CSS, images and other assets and rewrites links to local paths
Yes, <a>, <link> and <img>

--clone true and --links-file both switch seed mode on for you.

An earlier version of this post said a plain node index.js https://www.example.com "will crawl everything". It doesn't: without --seed or --clone it only processes the one page. If it doesn't exit by itself once the page is saved, press Ctrl+C.

Where the files go

Pages and assets are saved under ./www/ in the folder you run from (change it with --webroot), in a folder per host that mirrors the URL path:

plaintext
www/
  www.example.com/
    index.html          <- https://www.example.com/
    about/index.html    <- https://www.example.com/about/
    css/site.css
    images/logo.png

A URL ending in / becomes index.html. In clone mode, links in the saved HTML are rewritten to point at the local copies, so you can open the result in a browser (--change-link false turns that off).

Cloning a section, or including other hosts

To clone only part of a site, give a path prefix:

bash
node index.js --clone true --path-pattern /docs https://www.example.com/docs/

The path filter applies to everything the queue fetches, so stylesheets and images stored outside /docs are skipped too.

Assets often live on another host, such as a CDN. In clone mode, links to other hosts are dropped unless they match an --include pattern, and the queue only fetches hosts that match the seed or --pattern. So name the extra host in both:

bash
node index.js --clone true \
  --include "cdn\.example\.com" \
  --pattern "cdn\.example\.com" \
  https://www.example.com/

Two things to know about these patterns:

  • They are regular expressions matched against the host name only, not the full URL. Escape dots if you want them literal.
  • --pattern takes one value. If you repeat it, the last one wins. An earlier version of this post showed --pattern a --pattern b; use one expression with | instead: --pattern "trove\.example\.com|gold\.example2\.com". --include and --exclude do accept repeats.

How Redis is used, and how to start over

Every link the crawler finds is stored in Redis with a status (not crawled, in progress, done). That's what lets you stop a long crawl and run the same command again later: it carries on with the links that aren't done yet.

The flip side catches people out. Run the same crawl twice and the second run does almost nothing, because every link is already marked done. To start fresh, delete the crawler's keys.

By default all crawls share one namespace, tmp. With --han true ("host as namespace") the keys are named after the first URL's host instead, which keeps crawls of different sites apart and makes them easy to clear:

bash
# see what's there
redis-cli --scan --pattern 'www.example.com*'
# start that site from scratch (also removes its saved cookies)
redis-cli --scan --pattern 'www.example.com*' | xargs -r redis-cli del

# default namespace
redis-cli --scan --pattern 'tmp*' | xargs -r redis-cli del

The tmp* pattern matches any key starting with "tmp", so only run it against a Redis that isn't shared with other apps.

Options and what they actually do

The README lists the options. This table is what the current code does with them, which isn't always the same:

Option
Default
What it does
--seed true
false
Crawl from the given URL(s) and follow links on the same host(s)
--clone true
false
Seed mode plus assets and local link rewriting
--show-window true
false
Show the browser instead of running headless. Use it whenever you're writing actions
--webroot DIR
./www
Where pages and assets are saved
--han true
false
Use the first URL's host as the Redis namespace instead of tmp
--pattern REGEX
none
Extra host names the queue may visit. Single value
--path-pattern /path
none
Clone mode only: only fetch URLs under this path
--include REGEX
[]
Clone mode only: also collect links on hosts matching this
--exclude REGEX
[]
Non-clone modes only: skip links on hosts matching this
--actions-file PATH
none
JSON file of actions to run on each page (see below)
--processor NAME
none
Custom page processor from processors/, such as x or evernote
--links-file PATH
none
Text file of URLs, one per line. Turns on seed mode
--browser-wait-time MS
0 (means 30000)
The page-load timeout. Puppeteer waits until the network is nearly idle, up to this long. It is not an extra pause after loading
--wait-time MS
1200
Clone mode: pause between pages once more than 6 pages in a row come from the same host
--with-cookies true
false
Load cookies saved in Redis for the host before the page, and save the page's cookies after the actions run
--cookies-file PATH
none
JSON array of cookies to load (needs --with-cookies true)
--local-storage PATH
none
JSON object of localStorage keys to set after the page loads. Same ./ path rule as --actions-file
--viewonly true
false
Don't save the first page (useful for a login page). Later pages are saved
--with-curl true
false
Download non-HTML files (PDFs, images, CSS, JS) with curl instead of Node
--with-browser false
true
Skip Puppeteer and fetch everything over plain HTTP. No JavaScript runs
--level N
-1
Meant to limit depth. In the current code the check never runs, so don't rely on it. Use --path-pattern or --pattern to keep a crawl small
--next-crawl-wait-time S
31
Parsed but not used by the crawler at the moment

Boolean options are safest written with an explicit value (--clone true), especially right before the URL.

Logging in and clicking with actions.json

An actions file is a JSON array. Each entry is usually a condition plus a list of steps: if an element matching the if selector is on the page, run the then steps in order.

Here's a login, with a cookie banner handled first:

json
[
  {
    "if": ".cookie-consent",
    "then": [
      { "action": "click", "selector": "button.accept-cookies", "text": "Accept" }
    ]
  },
  {
    "if": "#login-form",
    "then": [
      { "action": "type", "selector": "input[name='username']", "value": "YOUR_USERNAME" },
      { "action": "type", "selector": "input[name='password']", "value": "YOUR_PASSWORD" },
      { "action": "click", "selector": "button[type='submit']" },
      { "action": "wait", "time": 3000 },
      { "action": "saveCookies" }
    ]
  }
]

Run it with the browser visible the first time, so you can see where it gets stuck:

bash
node index.js --show-window true --with-cookies true \
  --actions-file ./actions.json https://www.example.com/login

Write the path as ./actions.json or an absolute path. The file is loaded with Node's require(), so a bare actions.json is treated as a package name and fails with "Error reading actions file", and the crawl carries on without your actions. A ./ path is resolved from the tyo-crawler folder, not from where you are. (An earlier version of this post used the bare name.)

The action types the code supports:

Action
Fields
What it does
type
selector, value
Clears the input, then types the value
click
selector, optional text
Without text: clicks and waits for the page to navigate. With text: clicks the first matching element whose text contains it, without waiting
wait
time (ms), or selector (+ optional text)
Pauses, or waits for an element to appear
eval
selector, value
Sets the element's value directly
scroll
value (optional)
Scrolls to that position, or to the bottom
goto
url
Navigates to another page
reload
Reloads the page
saveCookies
Saves the current cookies to Redis for this host
setLocalStorage / setSessionStorage
key, value
Sets a storage key
screenshot
path (optional)
Saves a screenshot, screenshot.png by default
iframe
selector
Reads the content of an iframe (used by the Evernote processor)
evaluate / process
Grab the page HTML and pass it to the processor
hold
url
Only logs a message. It does not stop the crawl, whatever older docs say
exit
Closes the browser and ends the program

Any action with "on": false is skipped, which is handy for switching steps off while testing.

Besides if/then, an entry can be a repeat loop: it runs its steps over and over until an until selector appears, or until a process step fails failed_limit times in a row. That's how the X processor scrolls a timeline. The full example is in examples/x.example.json and explained in the X scraping post.

A few practical notes:

  • A plain click waits for a navigation. If the button only opens a menu or loads content in place, add a text field so the click doesn't wait for a page load that never comes.
  • Two-factor codes change. The repo's example types a fixed code. That's only a placeholder. In practice, log in once with --show-window true, enter the code yourself, let saveCookies store the session, and reuse it with --with-cookies true.
  • Keep credentials out of git. An actions file with a real password in it is a secret. Keep it outside the repo or add it to .gitignore. The saved cookies sit in Redis unencrypted, which is another reason to keep Redis on localhost.

Being a polite crawler

tyo-crawler does not read robots.txt, and outside clone mode it doesn't add a delay between pages beyond the time each page takes to load. That's fine for your own sites and small jobs. For anyone else's site:

  • Check the site's robots.txt and terms of use yourself before you start.
  • Keep the crawl narrow with --path-pattern or --pattern.
  • In clone mode, raise --wait-time (for example --wait-time 5000).
  • Don't crawl behind someone else's login unless you're allowed to.

Known gaps in the current code

These are worth knowing before you build anything on it:

  • Redis host and port: the crawler connects to localhost:6379. --dbhost and --dbport exist, but they aren't passed to the Redis client in a form it uses. For a remote Redis, forward the port (for example ssh -L 6379:localhost:6379 yourserver).
  • TLS checks are off for downloads. Files fetched outside the browser (and with --with-curl, which runs curl -k) don't verify certificates. Pages loaded in the browser are checked as normal.
  • --with-curl doesn't quote URLs. It builds a shell command, so a URL with & or a space in it breaks. Only use it on sites you trust.
  • Old user agent. Pages are requested with a Chrome 86 user-agent string, which some sites treat as a bot or an outdated browser.
  • README examples use node crawler.js. The entry point is node index.js.

When to use something else

You want to
Better fit
Mirror a static site
wget --mirror --convert-links --adjust-extension --page-requisites --no-parent URL
Grab one rendered page
A short Puppeteer or Playwright script
Run a large, long-lived crawl in Node.js
Crawlee, which has request queues, retries and proxy rotation built in
Crawl in Python
Scrapy (add scrapy-playwright for JavaScript-heavy pages)
Log in, click through a site and save what you see, with a resumable queue
tyo-crawler

tyo-crawler suits the last row: small to medium jobs where you need a real browser, a scripted login and the ability to stop and resume. If you're crawling to feed a RAG system, the open-source RAG tools comparison covers what to do with the pages once you have them.

Related

Comments (0)

Leave a Comment
Your email won't be published. We'll only use it to notify you of replies to your comment.
Loading comments...
Previous Article
post-thumb

Apr 13, 2025

How to Scrape Twitter (X) User's Tweets Using tyo-crawler

In this article, we will explore how to use tyo-crawler to scrape tweets from X.

Next Article
post-thumb

Mar 14, 2025

I Tried 9 Open-Source RAG Tools on My Own Documents: What Worked

I loaded my own PDFs into nine self-hosted RAG tools (Open WebUI, Verba, Onyx, LobeChat, RAGFlow, RAG Web UI, Kotaemon, Cognita and MaxKB) and asked them questions. Here is what they answered, where they went wrong, and which I'd use.

agico

We transform visions into reality. We specializes in crafting digital experiences that captivate, engage, and innovate. With a fusion of creativity and expertise, we bring your ideas to life, one pixel at a time. Let's build the future together.

Copyright ©  2026  TYO Lab · v0.0.39