pandas – how to get last n groups of a groupby object and combine them as a dataframe

  • A+
Category:Languages

How to get last 'n' groups after df.groupby() and combine them as a dataframe.

data = pd.read_sql_query(sql=sqlstr, con=sql_conn, index_col='SampleTime') grouped = data.groupby(data.index.date,sort=False) 

After doing grouped.ngroups i am getting total number of groups 277. I want to combine last 12 groups and generate a dataframe.

 


Pandas GroupBy objects are iterables. To extract the last n elements of an iterable, there's generally no need to create a list from the iterable and slice the last n elements. This will be memory-expensive.

Instead, you can use collections.deque and specify maxlen. Then use pd.concat to concatenate a sequence of dataframes.

from collections import deque from operator import itemgetter  grouped = data.groupby(data.index.date, sort=False) res = pd.concat(deque(map(itemgetter(1), grouped), maxlen=12)) 

As described in the collections docs:

Once a bounded length deque is full, when new items are added, a corresponding number of items are discarded from the opposite end.... They are also useful for tracking transactions and other pools of data where only the most recent activity is of interest.

Comment

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: