From Zero to Regex Hero
Extract & Parse URL-like Strings with a Single Regex
Table of Contents
- 🚀 Introduction
- 🔍 Extracting URLs from Text
- 🛳️ The 120+ Byte Regex
- 🧩 Breaking It Down Step by Step
- 🛠️ Parsing Example
- ☑️ Next Steps
- 📝 Summary
- 📚 Further Learning
TL;DR: Jump ahead to the 120+ Byte Regex.
🚀 Introduction
Extracting URLs from raw text can sometimes feel like playing a tedious game of whack-a-mole. Punctuation, parenthetical wrappers, and ambiguous formatting all conspire to frustrate your efforts. Whether you’re building a web scraper, data analyzer, or a chat application, accurately extracting URLs is essential.
In this post, we’ll tackle the problem head-on with a flexible, two-step approach. Our goal is to capture all potential URL-like strings first and then handle validation in a subsequent process.
💡 Note: This pattern is not for validating URLs! It’s intentionally permissive with punctuation and bad spelling.
🔍 Goal: Extract URLs from Text
When extracting URLs from raw text, a two-step approach is effective:
- Capture Everything URL-like: Cast a wide net to grab all strings that could be URLs. This is where our “120+ byte regex” shines.
- Validate: Once you’ve captured these candidates, use secondary checks (e.g., DNS resolution, comparison against known domains) to weed out invalid entries.
Visualizing the Challenge
Terms like extract and parse are often used interchangeably, however they refer to distinct processes. Extracting URLs involves identifying and capturing potential URLs from a larger body of text. Parsing, on the other hand, involves breaking down these URLs into their constituent parts.
When I mention parsing or ‘URL parts’, I’m referring to the following components:
Click to see a screenshot of RegEx101’s substring matching.
Before we get too deep into the regex, let’s use a visual tool to see how well my pattern captures many matches:
🛳️ The 120+ Byte Regex
Below is a concise regex designed to extract and parse URLs in a single shot. It supports various protocols, domains, paths, and optional query/fragment sections.
Don’t worry—we’ll break it down step by step!
const urlRegex = /([-.a-z0-9]+:\/{1,3})([^-\/\.[\](|)\s?][^`\/\s\]?]+)([-_a-z0-9!@$%^&*()=+;/~\.]*)[?]?([^#\s`?]*)[#]?([^#\s'"`\.,!]*)/gi;// Compatibility: ES5+
// Same pattern, split on newlines for readability:([-.a-z0-9]+:\/{1,3})([^-\/\.[\](|)\s?][^`\/\s\]?]+)([-_a-z0-9!@$%^&*()=+;/~\.]*)[?]?([^#\s`?]*)[#]?([^#\s'"`\.,!]*)Share the wildest regex’s you’ve encountered (OR authored) in the comments below! 🚀
🧩 Breaking It Down Step by Step
Let’s dissect the regex into its components to understand how it works:
1. Protocol (Group 1): ([-.a-z0-9]+:/{1,3})
- Purpose: Matches the Protocol part of the URL (e.g.,
http://,ftp://,custom-scheme://). Explanation:
[-.a-z0-9]+: Matches one or more lowercase letters, digits, hyphens, or periods (common in protocol schemes).:/{1,3}: Matches a colon followed by one to three slashes (:/,://, or:///).
2. Domain (Group 2): ([^-/.[](|)s?][^`/s]?]+)
- Purpose: Captures the domain or host part of the URL.
Explanation:
[^-/.[](|)\s?]: Matches any character except specified special characters and whitespace.[^`/\s]?]+: Matches one or more characters except backticks, slashes, whitespace, or closing square brackets.
3. Path (Group 3): ([-_a-z0-9!@$%^&*()=+;/~\.]*)
- Purpose: Matches the path component of the URL.
Explanation:
[-_a-z0-9!@$%^&()=+;/~.]: Matches zero or more URL-safe characters commonly found in paths.
4. Query (Group 4): [?]?([^#\s`?]*)
- Purpose: Optionally matches a query string, starting with any
?char. Explanation:
[?]?: Optionally matches a?. (The square brackets are not strictly necessary, however they are slightly more clear than the ultra terse double??. It also provides a visual parallel for the (similar) next matching group[#]?.)([^#\s`?]*): Matches zero or more characters that are not a hash, whitespace, backtick, or question mark.
5. Fragment (Group 5): [#]?([^#\s’”`.,!]*)
- Purpose: Optionally matches the fragment identifier starting with a
#. Explanation:
[#]?: Optionally matches a#.([^#\s’”`.,!]*): Matches zero or more characters that are not prohibited punctuation or whitespace.
🛠️ Parsing Example
Here’s how you can put this monster regex to work, with a bit of JavaScript:
const text = `Check this out: https://example.com/path?query=123#sectionAnd also (ftp://files.server.org/index).Plus a weird one: custom-scheme://host/param;weird^stuff`;
const urlRegex = /([-.a-z0-9]+:\/{1,3})([^-\/\.[\](|)\s?][^`\/\s\]?]+)([-_a-z0-9!@$%^&*()=+;/~\.]*)[?]?([^#\s`?]*)[#]?([^#\s'"`\.,!]*)/gi;
const matches = [ ...text.matchAll(urlRegex),].map((match) => match[0]);console.log("Extracted URLs:", matches);
const parts = [ ...text.matchAll(urlRegex),].map((match) => match.slice(1));console.log("Extracted Parts:", parts);[ "https://example.com/path?query=123#section", "ftp://files.server.org/index", "custom-scheme://host/param;weird^stuff"][ [ "https://", // Protocol "example.com", // Domain "/path", // Path "query=123", // Query "section" // Fragment ], [ "ftp://", // Protocol "files.server.org", // Domain "/index", // Path "", // Query "" // Fragment ], [ "custom-scheme://", // Protocol "host", // Domain "/param;weird^stuff", // Path "", // Query "" // Fragment ]]☑️ Next Steps
Depending on your use case, you might need to refine this regex or add more validation and post-processing steps.
Different Projects, Different Needs
Projects have varied requirements and security concerns:
- Web Scraping: Validate URLs to ensure they’re reachable and trustworthy.
- Data Processing: Extract URLs from user-generated content while ensuring safety.
- Data Analysis: Filter out duplicates or irrelevant links for research or marketing purposes.
- User-facing Applications: Automatically hyperlink URLs in chat apps or forums.
Post-Processing and Validation
After gathering potential URLs, apply additional checks:
- DNS Lookup: Verify that domains resolve.
- Safety Checks: Use services to check for malicious or phishing sites.
- Custom Rules: Apply project-specific filters (e.g., allowed TLDs, maximum URL length).
📝 Summary
Extracting semi-structured string data just might be the most satisfying part of regex mastery.
Here’s a recap of the key takeaways:
- Use a visual tool to write, test & understand your Regex patterns.
- Break down challenge into parts solve each part separately. In a sense, capture groups provide us figurative ‘trail markers’ for our regex.
- Use ‘loose’ match expressions, avoid strict spec conformance when doing data ingestion.
- Applying validation steps after the initial extraction is essential—always consider your project’s security and specific needs.
By following these steps, you can effectively extract any semi-structured string data, setting the foundation for further processing and validation.
📚 Further Learning
- Remember to play with a live demo on RegEx101.com!
- Original StackOverflow question, and a link to my answer right here.
- MDN Docs on Regular Expressions
- Advanced Regex Techniques: Explore lookaheads, lookbehinds, and other advanced patterns for more precise matching.
- RFC 3986 - URI Generic Syntax
