raccoon.dataframe module

DataFrame class

class raccoon.dataframe.DataFrame(data: dict[ColumnT, list[Any] | Any] | None = None, columns: Sequence[ColumnT] | None = None, index: Sequence[IndexT] | None = None, index_name: str | tuple | None = 'index', sort: bool | None = None)

Bases: Generic

DataFrame class. The raccoon DataFrame implements a simplified version of the pandas DataFrame with the key objective difference that the raccoon DataFrame is meant for use cases where the size of the DataFrame rows is expanding frequently. This is known to be slow with Pandas due to the use of numpy as the underlying data structure. Raccoon uses native lists as the underlying data structure which is quick to expand and grow the size. The DataFrame can be designated as sort, in which case the rows will be sort by index on construction, and then any addition of a new row will insert it into the DataFrame so that the index remains sort.

Parameters:

• data – (optional) dictionary of lists. The keys of the dictionary will be used for the column names and the lists will be used for the column data.

• columns – (optional) list of column names that will define the order

• index – (optional) list of index values. If None then the index will be integers starting with zero

• index_name – (optional) name for the index. Default is “index”

• sort – if True then DataFrame will keep the index sort. If True all index values must be of same type. If None then will default to True if no index is provided.

add(left_column: Any, right_column: Any, indexes: list[Any] | list[bool] | None = None) → list[Any]

Math helper method that adds element-wise two columns. If indexes are not None then will only perform the math on that sub-set of the columns.

Parameters:

• left_column – first column name

• right_column – second column name

• indexes – list of index values or list of booleans. If a list of booleans then the list must be the same length as the DataFrame

Returns:

list

append(data_frame: Self) → None

Append another DataFrame to this DataFrame. If the new data_frame has columns that are not in the current DataFrame then new columns will be created. All of the indexes in the data_frame must be different from the current indexes or will raise an error.

Parameters:

data_frame – DataFrame to append

Returns:

nothing

append_row(index: IndexT, values: dict[ColumnT, Any], new_cols: bool = True) → None

Appends a row of values to the end of the data. If there are new columns in the values and new_cols is True they will be added. Be very careful with this function as for sort DataFrames it will not enforce sort order. Use this only for speed when needed, be careful.

Parameters:

• index – value of the index

• values – dictionary of values

• new_cols – if True add new columns in values, if False ignore

Returns:

nothing

append_rows(indexes: list[IndexT], values: dict[ColumnT, list[Any]], new_cols: bool = True) → None

Appends rows of values to the end of the data. If there are new columns in the values and new_cols is True they will be added. Be very careful with this function as for sort DataFrames it will not enforce sort order. Use this only for speed when needed, be careful.

Parameters:

• indexes – list of indexes

• values – dictionary of values where the key is the column name and the value is a list

• new_cols – if True add new columns in values, if False ignore

Returns:

nothing

property columns: list[ColumnT]

property data: list[list[Any]]

delete_all_rows() → None

Deletes the contents of all rows in the DataFrame. This function is faster than delete_rows() to remove all information, and at the same time it keeps the container lists for the columns and index so if there is another object that references this DataFrame, like a ViewSeries, the reference remains intact.

Returns:

nothing

delete_columns(columns: Any | list[Any]) → None

Delete columns from the DataFrame

Parameters:

columns – list of columns to delete

Returns:

nothing

delete_rows(indexes: Any | list[Any] | list[bool]) → None

Delete rows from the DataFrame

Parameters:

indexes – either a list of values or list of booleans for the rows to delete

Returns:

nothing

divide(left_column: Any, right_column: Any, indexes: list[Any] | list[bool] | None = None) → list[Any]

Math helper method that divides element-wise two columns. If indexes are not None then will only perform the math on that sub-set of the columns.

Parameters:

• left_column – column name of dividend

• right_column – column name of divisor

• indexes – list of index values or list of booleans. If a list of booleans then the list must be the same length as the DataFrame

Returns:

list

equality(column: Any, indexes: list[Any] | list[bool] | None = None, value: Any = None) → list[bool]

Math helper method. Given a column and optional indexes will return a list of booleans on the equality of the value for that index in the DataFrame to the value parameter.

Parameters:

• column – column name to compare

• indexes – list of index values or list of booleans. If a list of booleans then the list must be the same length as the DataFrame

• value – value to compare

Returns:

list of booleans

classmethod from_json(json_string: str) → Self

Creates and return a DataFrame from a JSON of the type created by to_json.

Parameters:

json_string – JSON

Returns:

DataFrame

get(indexes: list[IndexT] | list[bool], columns: ColumnT, *, as_list: Literal[True], as_dict: Literal[False] = False) → list[Any]

get(indexes: list[IndexT] | list[bool], columns: ColumnT, as_list: Literal[False] = False, as_dict: Literal[False] = False) → DataFrame[IndexT, ColumnT]

get(indexes: None = None, columns: ColumnT | None = None, *, as_list: Literal[True], as_dict: Literal[False] = False) → list[Any]

get(indexes: None = None, columns: ColumnT | None = None, as_list: Literal[False] = False, as_dict: Literal[False] = False) → DataFrame[IndexT, ColumnT]

get(indexes: IndexT, columns: list[ColumnT] | list[bool] | None, *, as_list: Literal[False] = False, as_dict: Literal[True]) → dict[Any, Any]

get(indexes: IndexT, columns: list[ColumnT] | list[bool] | None, as_list: Literal[False] = False, as_dict: Literal[False] = False) → DataFrame[IndexT, ColumnT]

get(indexes: list[IndexT] | list[bool] | None = None, columns: list[ColumnT] | list[bool] | None = None, as_list: Literal[False] = False, as_dict: Literal[False] = False) → Self

get(indexes: IndexT, columns: ColumnT, as_list: Literal[False] = False, as_dict: Literal[False] = False) → Any

Given indexes and columns will return a sub-set of the DataFrame. This method will direct to the below methods based on what types are passed in for the indexes and columns. The type of the return is determined by the types of the parameters.

Parameters:

• indexes – index value, list of index values, or a list of booleans. If None then all indexes are used

• columns – column name or list of column names. If None then all columns are used

• as_list – if True then return the values as a list, if False return a DataFrame. This is only used if the get is for a single column

• as_dict – if True then return the values as a dictionary, if False return a DataFrame. This is only used if the get is for a single row

Returns:

either DataFrame, list, dict or single value. The return is a shallow copy

get_cell(index: IndexT, column: ColumnT) → Any

For a single index and column value return the value of the cell

Parameters:

• index – index value

• column – column name

Returns:

value

get_columns(index: IndexT, columns: list[ColumnT] | None = None, *, as_dict: Literal[True], as_namedtuple: Literal[True], name: str = 'raccoon', include_index: bool = True) → Never

get_columns(index: IndexT, columns: list[ColumnT] | None = None, *, as_dict: Literal[True], as_namedtuple: Literal[False] = False, name: str = 'raccoon', include_index: bool = True) → dict[Any, Any]

get_columns(index: IndexT, columns: list[ColumnT] | None = None, *, as_dict: Literal[False] = False, as_namedtuple: Literal[True], name: str = 'raccoon', include_index: bool = True) → tuple[Any, ...]

get_columns(index: IndexT, columns: list[ColumnT] | None = None, *, as_dict: Literal[False] = False, as_namedtuple: Literal[False] = False, name: str = 'raccoon', include_index: bool = True) → DataFrame[IndexT, ColumnT]

For a single index and list of column names return a DataFrame of the values in that index as either a dict, tuple-like namedtuple instance, or a DataFrame.

Parameters:

• index – single index value

• columns – list of column names

• as_dict – if True then return the result as a dictionary

• as_namedtuple – if True then return the result as a named tuple

• name – if as_namedtuple is True, this will be the name of the tuple

• include_index – if True then include the index value in the result

Returns:

DataFrame, dictionary, or tuple-like namedtuple instance

get_entire_column(column: ColumnT, as_list: Literal[True]) → list[Any]

get_entire_column(column: ColumnT, as_list: Literal[False] = False) → DataFrame[IndexT, ColumnT]

Shortcut method to retrieve a single column all rows. Since this is a common use case this method will be faster than the more general method.

Parameters:

• column – single column name

• as_list – if True return a list, if False return DataFrame

Returns:

DataFrame is as_list if False, a list if as_list is True

get_location(location: int, columns: list[ColumnT] | list[bool] | None = None, *, as_dict: Literal[True], as_namedtuple: Literal[True], name: str = 'raccoon', index: bool = True) → Never

get_location(location: int, columns: list[ColumnT] | list[bool] | None = None, *, as_dict: Literal[True], as_namedtuple: Literal[False] = False, name: str = 'raccoon', index: bool = True) → dict[Any, Any]

get_location(location: int, columns: list[ColumnT] | list[bool] | None = None, *, as_dict: Literal[False] = False, as_namedtuple: Literal[True], name: str = 'raccoon', index: bool = True) → tuple[Any, ...]

get_location(location: int, columns: list[ColumnT] | list[bool] | None = None, *, as_dict: Literal[False] = False, as_namedtuple: Literal[False] = False, name: str = 'raccoon', index: bool = True) → DataFrame[IndexT, ColumnT]

get_location(location: int, columns: ColumnT, *, as_dict: Literal[False] = False, as_namedtuple: Literal[False] = False, name: str = 'raccoon', index: bool = True) → Any

For an index location and either (1) list of columns return a DataFrame or dictionary of the values or (2) single column name and return the value of that cell. This is optimized for speed because it does not need to look up the index location with a search. Also, can accept relative indexing from the end of the DataFrame in standard python notation [-3, -2, -1]

Parameters:

• location – index location in standard python form of positive or negative number

• columns – list of columns, single column name, or None to include all columns

• as_dict – if True then return a dictionary

• as_namedtuple – if True then return the result as a named tuple

• name – if as_namedtuple is True, this will be the name of the tuple

• index – if True then include the index in the dictionary if as_dict=True

Returns:

DataFrame, dictionary, tuple-like namedtuple instance, or a single cell value

get_locations(locations: list[int], columns: Any | list[Any] | list[bool] | None = None, **kwargs: Any) → DataFrame[IndexT, ColumnT]

For list of locations and list of columns return a DataFrame of the values.

Parameters:

• locations – list of index locations

• columns – list of column names

• kwargs – will pass along these parameters to the get() method

Returns:

DataFrame

get_matrix(indexes: list[Any] | list[bool], columns: list[Any] | list[bool]) → DataFrame[IndexT, ColumnT]

For a list of indexes and list of columns return a DataFrame of the values.

Parameters:

• indexes – either a list of index values or a list of booleans with same length as all indexes

• columns – list of column names

Returns:

DataFrame

get_rows(indexes: list[IndexT] | list[bool], column: ColumnT, as_list: Literal[True]) → list[Any]

get_rows(indexes: list[IndexT] | list[bool], column: ColumnT, as_list: Literal[False] = False) → DataFrame[IndexT, ColumnT]

For a list of indexes and a single column name return the values of the indexes in that column.

Parameters:

• indexes – either a list of index values or a list of booleans with same length as all indexes

• column – single column name

• as_list – if True return a list, if False return DataFrame

Returns:

DataFrame is as_list if False, a list if as_list is True

get_slice(start_index: Any = None, stop_index: Any = None, columns: list[ColumnT] | list[bool] | None = None, *, as_dict: Literal[True]) → tuple[list[IndexT], dict[ColumnT, list[Any]]]

get_slice(start_index: Any = None, stop_index: Any = None, columns: list[ColumnT] | list[bool] | None = None, *, as_dict: Literal[False] = False) → DataFrame[IndexT, ColumnT]

For sorted DataFrames will return either a DataFrame or dict of all the rows where the index is greater than or equal to the start_index if provided and less than or equal to the stop_index if provided. If either the start or stop index is None then will include from the first or last element, similar to standard python slide of [:5] or [:5]. Both end points are considered inclusive.

Parameters:

• start_index – lowest index value to include, or None to start from the first row

• stop_index – highest index value to include, or None to end at the last row

• columns – list of column names to include, or None for all columns

• as_dict – if True then return a tuple of (list of index, dict of column names: list data values)

Returns:

DataFrame or tuple

head(rows: int) → Self

Return a DataFrame of the first N rows

Parameters:

rows – number of rows

Returns:

DataFrame

property index: list[IndexT]

Return a view of the index as a list. Because this is a view any change to the return list from this method will corrupt the DataFrame.

Returns:

list

property index_name: str | tuple | None

isin(column: Any, compare_list: list[Any]) → list[bool]

Returns a boolean list where each element is whether that element in the column is in the compare_list.

Parameters:

• column – single column name, does not work for multiple columns

• compare_list – list of items to compare to

Returns:

list of booleans

iterrows(index: bool = True) → Iterator[dict[Any, Any]]

Iterates over DataFrame rows as dictionary of the values. The index will be included.

Parameters:

index – if True include the index in the results

Returns:

dictionary

itertuples(index: bool = True, name: str = 'Raccoon') → Iterator[tuple[Any, ...]]

Iterates over DataFrame rows as tuple of the values.

Field names on the returned namedtuple instances are normalized to valid Python identifiers. Invalid characters are replaced with underscores, leading/trailing underscores are removed, names that start with a digit or match a Python keyword are prefixed, and duplicate normalized names are suffixed to keep them unique.

Parameters:

• index – if True then include the index

• name – name of the namedtuple

Returns:

tuple-like namedtuple instances

multiply(left_column: Any, right_column: Any, indexes: list[Any] | list[bool] | None = None) → list[Any]

Math helper method that multiplies element-wise two columns. If indexes are not None then will only perform the math on that sub-set of the columns.

Parameters:

• left_column – first column name

• right_column – second column name

• indexes – list of index values or list of booleans. If a list of booleans then the list must be the same length as the DataFrame

Returns:

list

print(index: bool = True, **kwargs: Any) → None

Print the contents of the DataFrame. This method uses the tabulate function from the tabulate package. Use the kwargs to pass along any arguments to the tabulate function.

Parameters:

• index – If True then include the indexes as a column in the output, if False ignore the index

• kwargs – Parameters to pass along to the tabulate function

Returns:

output of the tabulate function

rename_columns(rename_dict: dict[ColumnT, ColumnT]) → None

Renames the columns

Parameters:

rename_dict – dict where the keys are the current column names and the values are the new names

Returns:

nothing

reset_index(drop: bool = False) → None

Resets the index of the DataFrame to simple integer list and the index name to ‘index’. If drop is True then the existing index is dropped, if drop is False then the current index is made a column in the DataFrame with the index name the name of the column. If the index is a tuple multi-index then each element of the tuple is converted into a separate column. If the index name was ‘index’ then the column name will be ‘index_0’ to not conflict on print().

Parameters:

drop – if True then the current index is dropped, if False then index converted to columns

Returns:

nothing

select_index(compare: Any | tuple, result: Literal['boolean'] = 'boolean') → list[bool]

select_index(compare: IndexT | tuple[Any, ...], result: Literal['value']) → list[IndexT]

Finds the elements in the index that match the compare parameter and returns either a list of the values that match, or a boolean list the length of the index with True to each index that matches. If the indexes are tuples then the compare is a tuple where None in any field of the tuple will be treated as “*” and match all values.

Parameters:

• compare – value to compare as a singleton or tuple

• result – ‘boolean’ = returns a list of booleans, ‘value’ = returns a list of index values that match

Returns:

list of booleans or values

set(indexes: Any | list[Any] | list[bool] | None = None, columns: ColumnT | None = None, values: Any | list[Any] = None) → None

Given indexes and columns will set a sub-set of the DataFrame to the values provided. This method will direct to the below methods based on what types are passed in for the indexes and columns. If the indexes or columns contains values not in the DataFrame then new rows or columns will be added.

Parameters:

• indexes – indexes value, list of indexes values, or a list of booleans. If None then all indexes are used

• columns – columns name, if None then all columns are used. Currently, can only handle a single column or all columns

• values – value or list of values to set (index, column) to. If setting just a single row, then must be a dict where the keys are the column names. If a list then must be the same length as the indexes parameter, if indexes=None, then must be the same and length of DataFrame

Returns:

nothing

set_cell(index: IndexT, column: ColumnT, value: Any) → None

Sets the value of a single cell. If the index and/or column is not in the current index/columns then a new index and/or column will be created.

Parameters:

• index – index value

• column – column name

• value – value to set

Returns:

nothing

set_column(index: list[Any] | list[bool] | None = None, column: Any | None = None, values: Any | list[Any] = None) → None

Set a column to a single value or list of values. If any of the index values are not in the current indexes then a new row will be created.

Parameters:

• index – list of index values or list of booleans. If a list of booleans then the list must be the same length as the DataFrame

• column – column name

• values – either a single value or a list. The list must be the same length as the index list if the index list is values, or the length of the True values in the index list if the index list is booleans

Returns:

nothing

set_location(location, values, missing_to_none=False)

Sets the column values, as given by the keys of the values dict, for the row at location to the values of the values dict. If missing_to_none is False then columns not in the values dict will be left unchanged, if it is True then they are set to None. This method does not add new columns and raises an error.

Parameters:

• location – location

• values – dict of column names as keys and the value as the value to set the row for that column to

• missing_to_none – if True set any column missing in the values to None, otherwise leave unchanged

Returns:

nothing

set_locations(locations: list[int], column: ColumnT, values: list[Any] | Any) → None

For a list of locations and a column set the values.

Parameters:

• locations – list of index locations

• column – column name

• values – list of values or a single value

Returns:

nothing

set_row(index: IndexT, values: dict[ColumnT, Any] | Any) → None

Sets the values of the columns in a single row.

Parameters:

• index – index value

• values – dict with the keys as the column names and the values what to set that column to

Returns:

nothing

property sort: bool

sort_columns(column: Any, key: Callable[[Any], Any] | None = None, reverse: bool = False) → None

Sort the DataFrame by one of the columns. The sort modifies the DataFrame inplace. The key and reverse parameters have the same meaning as for the built-in sort() function.

Parameters:

• column – column name to use for the sort

• key – if not None then a function of one argument that is used to extract a comparison key from each list element

• reverse – if True then the list elements are sort as if each comparison were reversed.

Returns:

nothing

sort_index() → None

Sort the DataFrame by the index. The sort modifies the DataFrame inplace

Returns:

nothing

subtract(left_column: Any, right_column: Any, indexes: list[Any] | list[bool] | None = None) → list[Any]

Math helper method that subtracts element-wise two columns. If indexes are not None then will only perform the math on that sub-set of the columns.

Parameters:

• left_column – first column name

• right_column – name of column to subtract from the left_column

• indexes – list of index values or list of booleans. If a list of booleans then the list must be the same length as the DataFrame

Returns:

list

tail(rows: int) → Self

Return a DataFrame of the last N rows

Parameters:

rows – number of rows

Returns:

DataFrame

to_dict(index: bool = True, ordered: bool = False) → dict[Any, Any] | OrderedDict[Any, Any]

Returns a dict where the keys are the column names and the values are lists of the values for that column.

Parameters:

• index – If True then include the index in the dict with the index_name as the key

• ordered – If True then return an OrderedDict() to preserve the order of the columns in the DataFrame

Returns:

dict or OrderedDict()

to_json() → str

Returns a JSON of the entire DataFrame that can be reconstructed back with raccoon.from_json(input). Any object that cannot be serialized will be replaced with the representation of the object using repr(). In that instance the DataFrame will have a string representation in place of the object and will not reconstruct exactly.

Returns:

json string

to_list() → list[Any]

For a single column DataFrame returns a list of the values. Raises error if more than one column.

Returns:

list

validate_integrity() → None

Validate the integrity of the DataFrame. This checks that the indexes, column names and internal data are not corrupted. Will raise an error if there is a problem.

Returns:

nothing