Extracting the text starting with a character and ends with another into new column in Python

In this blog post, we will explore how to extract text from a dataset using regular expressions in Python. Specifically, we will focus on extracting the ID from a link that starts with “tt” and ends before “/”. We will use the pandas library to manipulate the dataset.

Understanding Regular Expressions

Regular expressions (regex) are a powerful tool for matching patterns in text. They allow us to search for specific sequences of characters in strings. In this example, we will use regex to extract the ID from the link.

A regex pattern consists of several elements:

• Quantifiers: {n,m} specifies the range of matches, n is the minimum number of matches, and m is the maximum number of matches.
• Character classes: [abc] matches any character inside the brackets.
• Metacharacters: . matches any single character, \w matches word characters (letters, numbers, and underscores), and \d matches digits.

Using Regex to Extract the ID

We can use the str.extract method in pandas to extract the ID from the link. The regex pattern we will use is:

r'(?:\.*/title/tt)(?P<ID>\d+)(?:/.*)'

Explanation of the regex pattern:

• (.*/title/tt) matches any characters (including none) before “tt”. This allows us to capture the part of the link that comes before the ID.
• (?P<ID>\d+) captures one or more digits into a group called “ID”.
• (?:/.*) matches any characters after the ID. This is optional and only included if we want to keep the original link.

Here’s an example code block that demonstrates how to use this regex pattern:

import pandas as pd

# Create a sample dataset
data = {
    'Link': ['http://www.imdb.com/title/tt0114709/?ref_=fn_tt_tt_1', 
             'http://www.imdb.com/title/tt0123456/', 
             'http://www.imdb.com/title/tt0079019/']
}

df = pd.DataFrame(data)

# Use regex to extract the ID
df['ID'] = df['Link'].str.extract(r'(?:\.*/title/tt)(?P<ID>\d+)(?:/.*)')

print(df)

Output:

Using str.replace to Modify the Link

If we want to modify the link by replacing the part that comes before “tt” and after “tt”, we can use the str.replace method. Here’s an example code block that demonstrates how to do this:

import pandas as pd

# Create a sample dataset
data = {
    'Link': ['http://www.imdb.com/title/tt0114709/?ref_=fn_tt_tt_1', 
             'http://www.imdb.com/title/tt0123456/', 
             'http://www.imdb.com/title/tt0079019/']
}

df = pd.DataFrame(data)

# Use regex to replace the part that comes before "tt" and after "tt"
df['Link'] = df['Link'].str.replace(r'(.*/title/tt)(\d+)(/.*)', r'\1\2\3/ \2')

print(df)

Output:

As you can see, the part that comes before “tt” and after “tt” has been replaced with a space followed by the original number.

Conclusion

In this blog post, we explored how to extract text from a dataset using regular expressions in Python. We used the pandas library to manipulate the dataset and demonstrated two different techniques: extracting the ID from the link using str.extract, and modifying the link using str.replace. Regular expressions are a powerful tool for matching patterns in text and can be used in a variety of applications, including data cleaning, validation, and extraction.

Last modified on 2024-04-19