Orchestrate Web Crawls with Scrapyd and Automated Data Enrichment

Build Challenge

Drag nodes onto the canvas, connect them in order, then grade or reveal the solution.

n8n

Make

Zapier

Activepieces

Power Automate

🗺️ Real Workflow Canvas

Actual n8n node layout · click nodes to inspect

🏗️ Guided Build Along

Step-by-step with config snippets

🎯 Sandbox Missions

Extend · modify · prove you understand

📝 Practice Questions

Test your understanding

🧩 Mock Builder

Rebuild from scratch · graded

🔬 Deep Execution Playback

Agent thoughts · data flow · real output

🟠 Make Scenario

Mapped from n8n

🔗

Webhooks

Custom webhook

🌐

HTTP

Make a Request

💻

Tools

Run Code

🔀

Router

Route

⚙

Tools

Set Variable

🌐

HTTP

Make a Request

🏗️ Guided Build — Make

Step-by-step

Create a new scenario

Click Create a new scenario in Make. Search for Webhooks and select Custom webhook as your trigger. Save immediately to generate the webhook URL.

💡 Give your scenario a descriptive name before saving — Make uses it in logs and error emails.

Send a test payload

Copy the webhook URL and POST a sample JSON payload to it. Make captures the structure and makes fields available for mapping in later modules.

💡 Click the data bubble on the webhook module after a test run to inspect the received data structure.

Add the OpenAI module

Click + after the webhook and search for OpenAI. Select Create a Completion. Map the input fields from your webhook payload into the prompt using the variable picker.

💡 Set Max Tokens and Temperature explicitly — Make does not set defaults, so leaving them blank can produce inconsistent results.

Chain output modules and activate

Add any output modules (HTTP, Google Sheets, NocoDB). Set the scenario schedule or leave it as instant. Toggle the scenario to Active.

💡 Run the scenario once manually before activating. Check the Operations counter to see how many Make operations each run consumes.

🎯 Sandbox Mission

Extend it on Make

Add an error handler route

Make has a built-in error handling system using the Router module. Add an error route from your most critical module and configure it to send a Slack or email alert when the scenario fails.

Right-click any module → Add error handler. Choose Rollback, Ignore, or Resume as the directive, then wire the error path to a notification module.

⚡ Zapier Scenario

Mapped from n8n

⚡

Trigger

Webhooks by Zapier

Catch Hook

TRIGGER

🌐

Action

Webhooks by Zapier

GET

💻

Action

Code by Zapier

Run JavaScript

🔀

Action

Filter by Zapier

Only Continue If

⏱️

Action

Delay by Zapier

Delay For

🏗️ Guided Build — Zapier

Step-by-step

Create a Zap with Webhooks trigger

Click Create Zap. In the trigger step choose Webhooks by Zapier and select Catch Hook. Click Continue and copy the webhook URL shown.

💡 Webhooks by Zapier requires a paid plan. Use a native app trigger like Google Sheets: New Row to stay on free.

Send a test webhook

POST a sample JSON payload to the webhook URL. Zapier captures the structure and makes all fields available as variables in later steps.

💡 Use Postman or reqbin.com to send a quick test — paste your webhook URL and send any JSON body.

Add an OpenAI action

Click the + button. Search for OpenAI (ChatGPT) and choose Send Prompt. Map the data from your trigger into the prompt using the variable picker.

💡 Zapier waits for the full OpenAI response with no streaming. For long prompts, increase Zap timeout in settings or split into sub-Zaps.

Add output steps and publish

Chain any output actions (Google Sheets: Update Row, Gmail: Send Email). Click Publish Zap and toggle it on.

💡 Test each action step individually using the Test step button before publishing — Zapier will not warn you about missing fields until runtime.

🎯 Sandbox Mission

Extend it on Zapier

Add a Filter step to control when the Zap runs

Add a Filter step between your trigger and the OpenAI action. Set conditions so the Zap only continues when specific fields are populated — this prevents unnecessary API calls.

Click the + icon and choose Filter by Zapier. Add a condition based on any field from your trigger step.

🧩 Activepieces Scenario

Mapped from n8n

🔗

Http

New Webhook

🌐

Http

Send HTTP Request

💻

Data Mapper

Run Code

🔀

Branches

Branch

⚙

Data Mapper

Map Fields

🌐

Http

Send HTTP Request

🏗️ Guided Build — Activepieces

Step-by-step

Add the Webhook trigger piece

Create a new flow. Click Add Trigger and search for HTTP. Select New Webhook Event. Copy the generated URL and save the flow.

💡 Activepieces keeps the same webhook URL until you delete the piece — safe to reuse across test runs.

Add the OpenAI piece

Click + after the trigger and search for OpenAI. Select Ask ChatGPT. In the prompt field, use the variable picker to reference fields from your webhook payload.

💡 The OpenAI piece version matters — newer versions support GPT-4o. Check the Connections panel to confirm which model your key has access to.

Map output fields

Add a Data Mapper piece after OpenAI to extract and reshape the AI response before passing it downstream.

💡 Use {{step.openai.choices[0].message.content}} in the expression editor to pull the text content from the ChatGPT response.

Add output pieces and activate

Add NocoDB, Google Sheets, or any output piece. Connect credentials in the Connections panel. Click Test flow then Enable.

💡 Activepieces is self-hostable — run your own instance for production so your webhook URLs never change.

🎯 Sandbox Mission

Extend it on Activepieces

Swap OpenAI for Anthropic Claude

Activepieces has pieces for both OpenAI and Anthropic. Swap the OpenAI piece for @activepieces/piece-anthropic and compare output quality. Adjust the system message to suit Claude's style.

Claude 3 Haiku is significantly faster and cheaper than GPT-4o for simple extraction tasks. Benchmark both on 10 real inputs before committing.

🔵 Power Automate Scenario

Mapped from n8n

🔗

HTTP

When a HTTP request is received

TRIGGER

🌐

HTTP

Invoke an HTTP endpoint

💻

Data Operation

Compose

🔀

Control

Condition

⏱️

Control

Delay

🏗️ Guided Build — Power Automate

Step-by-step

Create an Instant cloud flow

In Power Automate click Create and choose Instant cloud flow. Select When a HTTP request is received as the trigger. Click Save to generate the POST URL.

💡 The HTTP trigger URL only appears after the first save — the endpoint is provisioned server-side on save.

Parse the incoming JSON

Add Data Operation and select Parse JSON. Paste a sample payload into the Schema box and click Generate from sample. This maps fields for all downstream steps.

💡 If your payload shape changes later, regenerate the schema or Power Automate silently drops new fields with no error.

Call AI Builder

Add AI Builder and select Create text with GPT. In the Prompt field, combine your parsed JSON fields with a natural language instruction using the Dynamic content picker.

💡 AI Builder credits are separate from Power Automate credits. Check your Microsoft 365 license for AI Builder allocation before testing.

Add output connectors and save

Add connector actions for any outputs (Excel Online, SharePoint, Outlook). Click Save and use the Test button to send a live test run.

💡 Run the Flow Checker before saving — it catches missing required inputs and misconfigured connections before runtime.

🎯 Sandbox Mission

Extend it on Power Automate

Add a Teams notification for every AI output

Add Microsoft Teams → Post a message after your AI step. Format the AI response as an adaptive card and post it to a Teams channel for team visibility.

Power Automate has native Adaptive Card support for Teams. Use adaptivecards.io/designer to build the card JSON, then paste it into the flow.

🧠 LLM Compatibility

Recommended

GPT-4o / GPT-4o-mini

OpenAI

FastLow costJSON mode

Best balance of speed/quality for email drafting and summarisation.

Claude 3.5 Sonnet

Anthropic

Long contextSafe

Excellent for long documents, nuanced tone, and multi-turn support.

Gemini 1.5 Pro

Google

1M contextMultimodal

Use when processing large documents or needing image + text in one step.

Mistral 7B / Mixtral

Mistral AI

Open weightSelf-host

Run locally via Ollama for zero API cost. Best for simple classification and drafts.

Llama 3.1 70B

Meta via Ollama

FreeLocal

Strong open model for production quality replies. Requires GPU server.

All models work with the n8n OpenAI, Anthropic, Google Gemini, and Ollama nodes. Swap models without changing workflow logic — only credentials and model name change.

🏆 Challenge: Orchestrate Web Crawls with Scrapyd and Automated Data Enrichment — Prysio AI Academy

Recreate the core node chain for this workflow

Study the animated playback above, then build the key node sequence in the canvas below. Match the trigger and processing steps this workflow uses.

⚙️ When clicking ‘Execute workf ⚙️ job_list ⚙️ filter_job Connect them left → right

Triggers

🔗

Webhook

Trigger

📧

Read Email

Trigger

✍️

Generate Text

OpenAI / Claude

🏷️

Classify

AI Label

Output

📬

Send Email

Gmail / SMTP

📨

Send Message

Slack / Telegram

📤

Return Output

Response

Data

🔍

Extract Data

Parse fields

⚙️

Transform

Set / Code

↙️

Drag nodes from the left panel

Click another node to connect →

Orchestrate Web Crawls with Scrapyd and Automated Data Enrichment

📖 Overview

🏗️ Architecture

▶️ Animated Playback

Build Challenge

🗺️ Real Workflow Canvas

🏗️ Guided Build Along

🎯 Sandbox Missions

📝 Practice Questions

🧩 Mock Builder

🔬 Deep Execution Playback

🟠 Make Scenario

🏗️ Guided Build — Make

🎯 Sandbox Mission

⚡ Zapier Scenario

🏗️ Guided Build — Zapier

🎯 Sandbox Mission

🧩 Activepieces Scenario

🏗️ Guided Build — Activepieces

🎯 Sandbox Mission

🔵 Power Automate Scenario

🏗️ Guided Build — Power Automate

🎯 Sandbox Mission

📊 Live Stats

⚠️ Common Mistakes

🗺️ Active Quests

🏆 Achievements

Orchestrate Web Crawls with Scrapyd and Automated Data Enrichment

📖 Overview

🏗️ Architecture

▶️ Animated Playback

Build Challenge

🗺️ Real Workflow Canvas

🏗️ Guided Build Along

🎯 Sandbox Missions

📝 Practice Questions

🧩 Mock Builder

🔬 Deep Execution Playback

🟠 Make Scenario

🏗️ Guided Build — Make

🎯 Sandbox Mission

⚡ Zapier Scenario

🏗️ Guided Build — Zapier

🎯 Sandbox Mission

🧩 Activepieces Scenario

🏗️ Guided Build — Activepieces

🎯 Sandbox Mission

🔵 Power Automate Scenario

🏗️ Guided Build — Power Automate

🎯 Sandbox Mission

📊 Live Stats

⚠️ Common Mistakes

🗺️ Active Quests

🏆 Achievements

Related Workflows