Extracting Links from a Web Page Using Python

Web scraping, the process of extracting data from websites, is a common need in today’s data-driven world. Python, a versatile and powerful programming language, offers a wealth of libraries to facilitate this task. In this article, we’ll explore how to write Python code to extract links from a web page.

Prerequisites

To complete this project, you’ll need the following Python libraries:

• requests: Used for fetching web pages.
• BeautifulSoup: Employed for parsing HTML content and extracting specific elements.

You can install these libraries using the following commands:

pip install requests
pip install beautifulsoup4

Code Explanation

Our code takes a user-provided URL and extracts all the links (anchor tags) found on that web page. Here’s a detailed breakdown of the code:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin

def get_all_links(url):
    try:
        # Fetch the web page
        response = requests.get(url)

        # Proceed if the request is successful
        if response.status_code == 200:
            soup = BeautifulSoup(response.text, 'html.parser')

            # Find all the links
            links = []
            for link in soup.find_all('a'):
                href = link.get('href')
                if href and href.startswith('http'):
                    links.append((link.text.strip() if link.text else "No Text", href))

            return links
        else:
            print("Error: Unable to fetch the page. Status code:", response.status_code)
            return []
    except Exception as e:
        print("Error:", str(e))
        return []

def main():
    # Main URL
    main_url = input("Enter URL: ")

    # Get all links
    all_links = get_all_links(main_url)

    # Print the results
    if all_links:
        print("Links found on the page:")
        for link_text, link_url in all_links:
            print(f"{link_text} - {link_url}")
    else:
        print("No links found on the page.")

if __name__ == "__main__":
    main()

Don’t forget to review our code server/client in our previous article. 😉

Conclusion

This Python code provides a simple yet effective tool for extracting links from a web page. It can serve as a foundation for various applications, including data mining, automation, and web scraping. By customizing and expanding upon this code, you can perform more advanced data extraction and analysis in your own projects.

We hope this article has been helpful in your journey to web scraping with Python! If you have any questions or feedback, please don’t hesitate to reach out. Happy coding!