
Last Update: March 27, 2025 7 min read
BY
eric
tyo-crawler: Crawl, Clone and Log In to Websites with Puppeteer and Redis
Short version: tyo-crawler is a Node.js command-line crawler. It drives a real Chrome through Puppeteer, so it can read pages that only render with JavaScript and pages behind a login. It keeps its link queue in Redis, so a crawl can be stopped and picked up again. You run it like this:
git clone https://github.com/tyolab/tyo-crawler.git
cd tyo-crawler
npm install
# Redis must be running on localhost:6379 (see below)
node index.js --clone true https://www.example.com/
If the site is plain static HTML, you don't need a browser at all: wget --mirror is simpler and faster (see "When to use something else" near the end).
I wrote tyo-crawler. An older version of it, built on Nightmare.js, has been in my repo since 2023. In March 2025 I rebuilt it on Puppeteer: the browser was launching on 26 March, logging in to a page worked the same day, and cloning a website worked on 27 March. Since then I've used it with custom processors to collect tweets from X and to turn shared Evernote notes into Markdown. This post covers the crawler itself. Everything below is checked against the code in the repo as of its latest commit (December 2025). The README is behind the code in a few places, and I point those out.
What you need
- Node.js 18 or later. The repo uses Puppeteer 24, and
npm installdownloads a matching Chrome build into~/.cache/puppeteer. - Redis on
localhost:6379. The crawler connects on start-up, every time. - Optionally curl, if you use
--with-curl.
Starting Redis
Either install it from your package manager:
sudo apt-get install redis-server
sudo systemctl start redis-server
redis-cli ping # should print PONG
or use the Compose file in the repo's docker/ folder:
cd docker && docker compose up -d && cd ..
That Compose file publishes port 6379 on all interfaces with no password (ALLOW_EMPTY_PASSWORD=yes). On a laptop behind a router that's usually fine. On a server with a public IP, anyone can then read and change your Redis, including the login cookies the crawler stores there. Bind it to localhost instead:
ports:
- "127.0.0.1:6379:6379"
If the bitnami/redis image won't pull for you, the official redis image works the same way for this purpose (image: redis:7, and mount the data folder at /data instead of /bitnami/redis/data).
The three ways to run it
What the crawler does depends mainly on two switches, --seed and --clone:
--clone true and --links-file both switch seed mode on for you.
An earlier version of this post said a plain node index.js https://www.example.com "will crawl everything". It doesn't: without --seed or --clone it only processes the one page. If it doesn't exit by itself once the page is saved, press Ctrl+C.
Where the files go
Pages and assets are saved under ./www/ in the folder you run from (change it with --webroot), in a folder per host that mirrors the URL path:
www/
www.example.com/
index.html <- https://www.example.com/
about/index.html <- https://www.example.com/about/
css/site.css
images/logo.png
A URL ending in / becomes index.html. In clone mode, links in the saved HTML are rewritten to point at the local copies, so you can open the result in a browser (--change-link false turns that off).
Cloning a section, or including other hosts
To clone only part of a site, give a path prefix:
node index.js --clone true --path-pattern /docs https://www.example.com/docs/
The path filter applies to everything the queue fetches, so stylesheets and images stored outside /docs are skipped too.
Assets often live on another host, such as a CDN. In clone mode, links to other hosts are dropped unless they match an --include pattern, and the queue only fetches hosts that match the seed or --pattern. So name the extra host in both:
node index.js --clone true \
--include "cdn\.example\.com" \
--pattern "cdn\.example\.com" \
https://www.example.com/
Two things to know about these patterns:
- They are regular expressions matched against the host name only, not the full URL. Escape dots if you want them literal.
--patterntakes one value. If you repeat it, the last one wins. An earlier version of this post showed--pattern a --pattern b; use one expression with|instead:--pattern "trove\.example\.com|gold\.example2\.com".--includeand--excludedo accept repeats.
How Redis is used, and how to start over
Every link the crawler finds is stored in Redis with a status (not crawled, in progress, done). That's what lets you stop a long crawl and run the same command again later: it carries on with the links that aren't done yet.
The flip side catches people out. Run the same crawl twice and the second run does almost nothing, because every link is already marked done. To start fresh, delete the crawler's keys.
By default all crawls share one namespace, tmp. With --han true ("host as namespace") the keys are named after the first URL's host instead, which keeps crawls of different sites apart and makes them easy to clear:
# see what's there
redis-cli --scan --pattern 'www.example.com*'
# start that site from scratch (also removes its saved cookies)
redis-cli --scan --pattern 'www.example.com*' | xargs -r redis-cli del
# default namespace
redis-cli --scan --pattern 'tmp*' | xargs -r redis-cli del
The tmp* pattern matches any key starting with "tmp", so only run it against a Redis that isn't shared with other apps.
Options and what they actually do
The README lists the options. This table is what the current code does with them, which isn't always the same:
Boolean options are safest written with an explicit value (--clone true), especially right before the URL.
Logging in and clicking with actions.json
An actions file is a JSON array. Each entry is usually a condition plus a list of steps: if an element matching the if selector is on the page, run the then steps in order.
Here's a login, with a cookie banner handled first:
[
{
"if": ".cookie-consent",
"then": [
{ "action": "click", "selector": "button.accept-cookies", "text": "Accept" }
]
},
{
"if": "#login-form",
"then": [
{ "action": "type", "selector": "input[name='username']", "value": "YOUR_USERNAME" },
{ "action": "type", "selector": "input[name='password']", "value": "YOUR_PASSWORD" },
{ "action": "click", "selector": "button[type='submit']" },
{ "action": "wait", "time": 3000 },
{ "action": "saveCookies" }
]
}
]
Run it with the browser visible the first time, so you can see where it gets stuck:
node index.js --show-window true --with-cookies true \
--actions-file ./actions.json https://www.example.com/login
Write the path as ./actions.json or an absolute path. The file is loaded with Node's require(), so a bare actions.json is treated as a package name and fails with "Error reading actions file", and the crawl carries on without your actions. A ./ path is resolved from the tyo-crawler folder, not from where you are. (An earlier version of this post used the bare name.)
The action types the code supports:
Any action with "on": false is skipped, which is handy for switching steps off while testing.
Besides if/then, an entry can be a repeat loop: it runs its steps over and over until an until selector appears, or until a process step fails failed_limit times in a row. That's how the X processor scrolls a timeline. The full example is in examples/x.example.json and explained in the X scraping post.
A few practical notes:
- A plain click waits for a navigation. If the button only opens a menu or loads content in place, add a
textfield so the click doesn't wait for a page load that never comes. - Two-factor codes change. The repo's example types a fixed code. That's only a placeholder. In practice, log in once with
--show-window true, enter the code yourself, letsaveCookiesstore the session, and reuse it with--with-cookies true. - Keep credentials out of git. An actions file with a real password in it is a secret. Keep it outside the repo or add it to
.gitignore. The saved cookies sit in Redis unencrypted, which is another reason to keep Redis on localhost.
Being a polite crawler
tyo-crawler does not read robots.txt, and outside clone mode it doesn't add a delay between pages beyond the time each page takes to load. That's fine for your own sites and small jobs. For anyone else's site:
- Check the site's
robots.txtand terms of use yourself before you start. - Keep the crawl narrow with
--path-patternor--pattern. - In clone mode, raise
--wait-time(for example--wait-time 5000). - Don't crawl behind someone else's login unless you're allowed to.
Known gaps in the current code
These are worth knowing before you build anything on it:
- Redis host and port: the crawler connects to
localhost:6379.--dbhostand--dbportexist, but they aren't passed to the Redis client in a form it uses. For a remote Redis, forward the port (for examplessh -L 6379:localhost:6379 yourserver). - TLS checks are off for downloads. Files fetched outside the browser (and with
--with-curl, which runscurl -k) don't verify certificates. Pages loaded in the browser are checked as normal. --with-curldoesn't quote URLs. It builds a shell command, so a URL with&or a space in it breaks. Only use it on sites you trust.- Old user agent. Pages are requested with a Chrome 86 user-agent string, which some sites treat as a bot or an outdated browser.
- README examples use
node crawler.js. The entry point isnode index.js.
When to use something else
tyo-crawler suits the last row: small to medium jobs where you need a real browser, a scripted login and the ability to stop and resume. If you're crawling to feed a RAG system, the open-source RAG tools comparison covers what to do with the pages once you have them.
Related
- Scrape an X (Twitter) user's tweets with tyo-crawler: actions with a
repeatloop and a custom processor - Convert Evernote shared notes to Markdown with tyo-crawler: the
iframeaction and the Evernote processor - Migrating a website from MODX to Next.js: where a site copy is the starting point









Comments (0)
Leave a Comment