In the world of data analysis, Pandas has emerged as a powerful tool, often referred to as the “Swiss Army knife” for data manipulation and analysis. Its intelligence is not just in its ability to handle large datasets efficiently but also in its versatility and the wide range of applications it supports. Let’s delve into 20 real-life examples that showcase the intelligence of Pandas.
1. Data Cleaning and Preprocessing
Example: A financial institution uses Pandas to clean and preprocess their transactional data before performing any analysis. Pandas helps in handling missing values, duplicate entries, and data type conversions.
import pandas as pd
data = {'Transaction ID': [1, 2, 3, 4, 5], 'Amount': [100, 200, None, 400, 500], 'Date': ['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04', '2021-01-05']}
df = pd.DataFrame(data)
df['Amount'].fillna(0, inplace=True)
df['Date'] = pd.to_datetime(df['Date'])
print(df)
2. Time Series Analysis
Example: An e-commerce company uses Pandas for time series analysis to understand customer buying patterns over different time intervals.
import pandas as pd
data = {'Date': ['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04', '2021-01-05'], 'Sales': [100, 150, 200, 250, 300]}
df = pd.DataFrame(data)
df['Date'] = pd.to_datetime(df['Date'])
df.set_index('Date', inplace=True)
print(df.resample('M').sum())
3. Data Aggregation
Example: A marketing team uses Pandas to aggregate sales data by region and product category to identify top-performing segments.
import pandas as pd
data = {'Region': ['North', 'South', 'East', 'West', 'North'], 'Category': ['A', 'B', 'A', 'B', 'A'], 'Sales': [100, 150, 200, 250, 300]}
df = pd.DataFrame(data)
result = df.groupby(['Region', 'Category']).sum()
print(result)
4. Data Merging and Joining
Example: A research team merges different datasets from various sources using Pandas to combine and analyze the data.
import pandas as pd
data1 = {'ID': [1, 2, 3], 'Name': ['Alice', 'Bob', 'Charlie']}
data2 = {'ID': [1, 2, 3], 'Age': [25, 30, 35]}
df1 = pd.DataFrame(data1)
df2 = pd.DataFrame(data2)
result = pd.merge(df1, df2, on='ID')
print(result)
5. Data Visualization
Example: A data scientist uses Pandas in conjunction with Matplotlib to create visualizations of sales data over time.
import pandas as pd
import matplotlib.pyplot as plt
data = {'Date': ['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04', '2021-01-05'], 'Sales': [100, 150, 200, 250, 300]}
df = pd.DataFrame(data)
df['Date'] = pd.to_datetime(df['Date'])
df.set_index('Date', inplace=True)
df.plot()
plt.show()
6. Text Analysis
Example: A content analysis team uses Pandas to analyze customer reviews and extract sentiment scores.
import pandas as pd
data = {'Review': ['Great product!', 'Not what I expected.', 'Excellent service!', 'Terrible experience.', 'Average quality.']}
df = pd.DataFrame(data)
df['Sentiment'] = df['Review'].apply(lambda x: 'Positive' if 'great' in x or 'excellent' in x else ('Negative' if 'not' in x or 'terrible' in x else 'Neutral'))
print(df)
7. Financial Analysis
Example: A hedge fund uses Pandas to analyze stock market data, calculate returns, and identify investment opportunities.
import pandas as pd
data = {'Date': ['2021-01-01', '2021-01-02', '2021-01-03', '2021-01-04', '2021-01-05'], 'Stock Price': [100, 102, 101, 103, 105]}
df = pd.DataFrame(data)
df['Return'] = df['Stock Price'].pct_change()
print(df)
8. Machine Learning Data Preparation
Example: A data scientist uses Pandas to prepare data for machine learning models, handling missing values, encoding categorical variables, and scaling numerical features.
import pandas as pd
from sklearn.preprocessing import StandardScaler
data = {'Feature1': [1, 2, 3, 4, 5], 'Feature2': ['A', 'B', 'C', 'D', 'E']}
df = pd.DataFrame(data)
df['Feature1'] = df['Feature1'].fillna(df['Feature1'].mean())
df['Feature2'] = pd.get_dummies(df['Feature2'])
scaler = StandardScaler()
df[['Feature1']] = scaler.fit_transform(df[['Feature1']])
print(df)
9. Social Media Analysis
Example: A marketing agency uses Pandas to analyze social media data, extracting insights from user comments and engagement metrics.
import pandas as pd
data = {'User': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'], 'Comment': ['Great product!', 'Not what I expected.', 'Excellent service!', 'Terrible experience.', 'Average quality.'], 'Likes': [50, 20, 100, 10, 30]}
df = pd.DataFrame(data)
df['Sentiment'] = df['Comment'].apply(lambda x: 'Positive' if 'great' in x or 'excellent' in x else ('Negative' if 'not' in x or 'terrible' in x else 'Neutral'))
print(df)
10. Web Scraping
Example: A data analyst uses Pandas to scrape data from a website, such as product prices and reviews, and analyze the data for trends.
import pandas as pd
import requests
from bs4 import BeautifulSoup
url = 'https://example.com/products'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
data = []
for product in soup.find_all('div', class_='product'):
name = product.find('h2', class_='name').text
price = product.find('span', class_='price').text
data.append({'Name': name, 'Price': price})
df = pd.DataFrame(data)
print(df)
11. Geospatial Analysis
Example: An urban planner uses Pandas to analyze spatial data, such as population density and land use, to plan city development.
import pandas as pd
data = {'Location': ['A', 'B', 'C', 'D', 'E'], 'Population': [1000, 2000, 1500, 2500, 3000], 'Land Use': ['Residential', 'Commercial', 'Industrial', 'Residential', 'Commercial']}
df = pd.DataFrame(data)
df['Population Density'] = df['Population'] / df['Land Use'].apply(lambda x: 1 if x == 'Residential' else 0.5)
print(df)
12. Event Stream Processing
Example: A real-time analytics platform uses Pandas to process and analyze event streams, such as user interactions on a website.
import pandas as pd
data = {'Timestamp': ['2021-01-01 12:00:00', '2021-01-01 12:01:00', '2021-01-01 12:02:00', '2021-01-01 12:03:00', '2021-01-01 12:04:00'], 'Event': ['Click', 'Click', 'Scroll', 'Click', 'Exit']}
df = pd.DataFrame(data)
df['Timestamp'] = pd.to_datetime(df['Timestamp'])
df.set_index('Timestamp', inplace=True)
print(df.resample('1T').count())
13. Sentiment Analysis
Example: A social media monitoring tool uses Pandas to analyze customer sentiment from tweets and posts, providing insights into brand perception.
import pandas as pd
data = {'Tweet': ['I love this product!', 'This is the worst thing I have ever bought.', 'Not sure about this product.', 'Absolutely amazing!', 'Terrible experience.']}
df = pd.DataFrame(data)
df['Sentiment'] = df['Tweet'].apply(lambda x: 'Positive' if 'love' in x or 'amazing' in x else ('Negative' if 'worst' in x or 'terrible' in x else 'Neutral'))
print(df)
14. Market Basket Analysis
Example: A retail company uses Pandas to perform market basket analysis, identifying products that are frequently purchased together.
import pandas as pd
data = {'Transaction ID': [1, 2, 3, 4, 5], 'Product 1': [1, 2, 1, 3, 1], 'Product 2': [2, 3, 2, 1, 2], 'Product 3': [3, 1, 3, 2, 3]}
df = pd.DataFrame(data)
result = df.groupby(['Transaction ID']).apply(lambda x: x['Product 1'].value_counts()).unstack(fill_value=0)
print(result)
15. Network Analysis
Example: A network engineer uses Pandas to analyze network traffic data, identifying patterns and anomalies.
import pandas as pd
data = {'Source': ['A', 'B', 'C', 'D', 'E'], 'Destination': ['B', 'C', 'A', 'E', 'D'], 'Traffic': [100, 150, 200, 250, 300]}
df = pd.DataFrame(data)
result = df.groupby(['Source', 'Destination']).sum()
print(result)
16. Event Logging
Example: A software development team uses Pandas to analyze event logs from their application, identifying and troubleshooting errors.
import pandas as pd
data = {'Timestamp': ['2021-01-01 12:00:00', '2021-01-01 12:01:00', '2021-01-01 12:02:00', '2021-01-01 12:03:00', '2021-01-01 12:04:00'], 'Event': ['Error', 'Warning', 'Info', 'Error', 'Warning']}
df = pd.DataFrame(data)
df['Timestamp'] = pd.to_datetime(df['Timestamp'])
df.set_index('Timestamp', inplace=True)
print(df.resample('1T').count())
17. Predictive Analytics
Example: A financial institution uses Pandas to analyze historical credit card transaction data, predicting the likelihood of default.
import pandas as pd
from sklearn.linear_model import LogisticRegression
data = {'Amount': [100, 200, 300, 400, 500], 'Credit Score': [700, 720, 710, 730, 740], 'Default': [0, 1, 0, 1, 0]}
df = pd.DataFrame(data)
model = LogisticRegression()
model.fit(df[['Amount', 'Credit Score']], df['Default'])
print(model.predict([[150, 715]]))
18. Text Mining
Example: A content provider uses Pandas to analyze customer feedback, extracting keywords and identifying common themes.
import pandas as pd
data = {'Review': ['I love this book!', 'This book is amazing!', 'Not what I expected.', 'Absolutely terrible!', 'Not my cup of tea.']}
df = pd.DataFrame(data)
df['Keywords'] = df['Review'].apply(lambda x: 'love' if 'love' in x else ('amazing' if 'amazing' in x else ('not' if 'not' in x else 'terrible')))
print(df)
19. Sentiment Analysis on Social Media
Example: A marketing team uses Pandas to analyze social media data, extracting sentiment scores and identifying key influencers.
import pandas as pd
data = {'User': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'], 'Tweet': ['I love this brand!', 'This brand is terrible.', 'Absolutely love this brand!', 'Not a fan of this brand.', 'Hate this brand.'], 'Likes': [50, 20, 100, 10, 30]}
df = pd.DataFrame(data)
df['Sentiment'] = df['Tweet'].apply(lambda x: 'Positive' if 'love' in x or 'absolutely' in x else ('Negative' if 'terrible' in x or 'hate' in x else 'Neutral'))
print(df)
20. Event Stream Analysis
Example: A real-time analytics platform uses Pandas to analyze event streams, such as user interactions on a website, and identify patterns and anomalies.
import pandas as pd
data = {'Timestamp': ['2021-01-01 12:00:00', '2021-01-01 12:01:00', '2021-01-01 12:02:00', '2021-01-01 12:03:00', '2021-01-01 12:04:00'], 'Event': ['Click', 'Click', 'Scroll', 'Click', 'Exit']}
df = pd.DataFrame(data)
df['Timestamp'] = pd.to_datetime(df['Timestamp'])
df.set_index('Timestamp', inplace=True)
print(df.resample('1T').count())
These examples showcase the intelligence of Pandas in handling a wide range of data analysis tasks. From data cleaning and preprocessing to advanced analytics and predictive modeling, Pandas continues to be a go-to tool for data professionals worldwide.