26. Pandas DataFrame

By Bernd Klein. Last modified: 21 Feb 2024.

The underlying idea of a DataFrame is based on spreadsheets. We can see the data structure of a DataFrame as tabular and spreadsheet-like. A DataFrame logically corresponds to a "sheet" of an Excel document. A DataFrame has both a row and a column index.

Like a spreadsheet or Excel sheet, a DataFrame object contains an ordered collection of columns. Each column consists of a unique data typye, but different columns can have different types, e.g. the first column may consist of integers, while the second one consists of boolean values and so on.

Here's a brief introduction to key concepts related to DataFrames:

• Rows and Columns: A DataFrame consists of rows and columns. Rows are indexed by default starting from 0, while columns have labels that identify them. The labels are typically strings but can also be integers.

• Data Types: Each column in a DataFrame can have its own data type. Common data types include integers, floats, strings, datetime objects, and categorical data.

• Indexing and Selection: DataFrames support various methods for indexing and selecting data, such as indexing by label, integer-location based indexing, boolean indexing, and more.

• Operations: DataFrames support a wide range of operations, including arithmetic operations, element-wise operations, aggregation functions (e.g., mean, sum), and statistical operations.

• Missing Data Handling: DataFrames provide methods for handling missing data, such as dropping missing values or filling them with a specified value.

• Merging and Concatenation: DataFrames can be merged or concatenated to combine data from different sources, either by rows or columns.

• Grouping and Aggregation: DataFrames support grouping operations, allowing you to group data based on one or more columns and perform aggregation functions on each group.

• Visualization: DataFrames can be easily visualized using built-in plotting functions in pandas or by integrating with other visualization libraries like Matplotlib or Seaborn.

Overall, DataFrames are a powerful tool for data analysis and manipulation, providing a flexible and intuitive way to work with structured data in Python.

Connection between DataFrames and Series Objects

There is a close connection between the DataFrames and the Series of Pandas. A DataFrame can be seen as a concatenation of Series, each Series having the same index, i.e. the index of the DataFrame.

We will demonstrate this in the following example.

We define the following three Series:

import pandas as pd

years = range(2014, 2018)

shop1 = pd.Series([2409.14, 2941.01, 3496.83, 3119.55], index=years)
shop2 = pd.Series([1203.45, 3441.62, 3007.83, 3619.53], index=years)
shop3 = pd.Series([3412.12, 3491.16, 3457.19, 1963.10], index=years)

What happens, if we concatenate these "shop" Series? Pandas provides a concat function for this purpose:

pd.concat([shop1, shop2, shop3])

2014    2409.14
2015    2941.01
2016    3496.83
2017    3119.55
2014    1203.45
2015    3441.62
2016    3007.83
2017    3619.53
2014    3412.12
2015    3491.16
2016    3457.19
2017    1963.10
dtype: float64

This result is not what we have intended or expected. The reason is that concat used 0 as the default for the axis parameter. Let's do it with "axis=1":

shops_df = pd.concat([shop1, shop2, shop3], axis=1)
shops_df

In this example, each column in shops_df represents a shop, and each cell contains the corresponding value from the original shop Series object. Each column is still a Pandas Series object, with its own index and values, whereas the whole structure is now a DataFrame object:

print(type(shops_df))
print(type(shops_df[0]))

<class 'pandas.core.frame.DataFrame'>
<class 'pandas.core.series.Series'>

Let's do some fine sanding by giving names to the columns:

cities = ["Zürich", "Winterthur", "Freiburg"]
shops_df.columns = cities 
print(shops_df)

# alternative way: give names to series: