Implement Concurrent Web Crawler

Given a website, design and implement a web crawler that discovers all URLs on this website.
You need to code this out.
(Programming Language: No limit)
def get_urls_from_page(url: str) -> List[str]:

Follow up 1: How would you improve the crawler's efficiency for large-scale domains? open discussion

Follow up 2: Implement an async version.
...




Get one-to-one training from Google Facebook engineers

Top-notch Professionals

Learn from Facebook and Google senior engineers interviewed 100+ candidates.
Most recent interview questions and system design topics gathered from aonecode alumnus.
One-to-one online classes. Get feedbacks from real interviewers.

Customized Private Class

Already a coding expert? - Advance straight to hard interview topics of your interest.
New to the ground? - Develop basic coding skills with your own designated mentor.
Days before interview? - Focus on most important problems in target company question bank.