Evals we should create

[aarondr77]

Mito AI Chat

• Very Large Active Code Cells. Active code cell was very large, (it contained a mapping from each country to its continent, generated by the AI), then when he asked it to make a change to the final line of that code cell that actually applies the mapping, instead of returning the entire mapping with the edit, it just returned the new edit.
• Empty kernels When the user has only ran import mitosheet and asks the Ai to do some edits to a dataframe. It should not revert to using example dataframes or hallucinate a dataframe called mitosheet
• Long chats with specific instructions Its having a really hard time continuing to follow his instructions to use the yahoo finance. Each time he goes to fix a problem in the cell, it reverts to the method that does not use the yfinance. He asked it once to use yfinance in second message and it should know to continue using that. Maybe this is because he has a long chat running and its having a hard time paying attention to the right pieces.
• Sending messages like hi or thanks should have reasonable responses
• Sending messages like what can you help me with? Do you know about my data etc should have reasonable responses

In Line Code Completer

• Don't return the prefix

  See example

  import pandas as pd
  nba_players_df = pd.DataFrame({
      'player_name': [
          'LeBron James', 'Kevin Durant', 'Stephen Curry', 'Giannis Antetokounmpo',
          'Kawhi Leonard', 'James Harden', 'Luka Dončić', 'Damian Lillard',
          'Joel Embiid', 'Nikola Jokić', 'Anthony Davis', 'Chris Paul',
          'Jayson Tatum', 'Zion Williamson', 'Devin Booker'
      ],
      'team': [
          'Los Angeles Lakers', 'Brooklyn Nets', 'Golden State Warriors', 'Milwaukee Bucks',
          'Los Angeles Clippers', 'Brooklyn Nets', 'Dallas Mavericks', 'Portland Trail Blazers',
          'Philadelphia 76ers', 'Denver Nuggets', 'Los Angeles Lakers', 'Phoenix Suns',
          'Boston Celtics', 'New Orleans Pelicans', 'Phoenix Suns'
      ],
      'position': [
          'SF', 'PF', 'PG', 'PF',
          'SF', 'SG', 'PG', 'PG',
          'C', 'C', 'PF', 'PG',
          'SF', 'PF', 'SG'
      ],
      'points_per_game': [
          25.0, 27.0, 29.0, 28.0,
          24.0, 25.0, 28.0, 27.0,
          33.0, 26.0, 22.0, 18.0,
          26.0, 22.0, 25.0
      ]
  })
  
  
  lakers_players = nba_players_df[nba_players_df['team'] == 'Los Angeles Lakers']
  nets_players = nba_players_df[nba_players_df['team'] == 'Brooklyn Nets']
  warriors_players = nba_players_df[nba_players_df['team'] == 'Golden State Warriors']
  buck

  Repeated the entire last line of the prefix

  

  Repeated the last character of the prefix

  

• Calling user defined function

  See example

  import pandas as pd
  nba_players_df = pd.DataFrame({
      'player_name': [
          'LeBron James', 'Kevin Durant', 'Stephen Curry', 'Giannis Antetokounmpo',
          'Kawhi Leonard', 'James Harden', 'Luka Dončić', 'Damian Lillard',
          'Joel Embiid', 'Nikola Jokić', 'Anthony Davis', 'Chris Paul',
          'Jayson Tatum', 'Zion Williamson', 'Devin Booker'
      ],
      'team': [
          'Los Angeles Lakers', 'Brooklyn Nets', 'Golden State Warriors', 'Milwaukee Bucks',
          'Los Angeles Clippers', 'Brooklyn Nets', 'Dallas Mavericks', 'Portland Trail Blazers',
          'Philadelphia 76ers', 'Denver Nuggets', 'Los Angeles Lakers', 'Phoenix Suns',
          'Boston Celtics', 'New Orleans Pelicans', 'Phoenix Suns'
      ],
      'position': [
          'SF', 'PF', 'PG', 'PF',
          'SF', 'SG', 'PG', 'PG',
          'C', 'C', 'PF', 'PG',
          'SF', 'PF', 'SG'
      ],
      'points_per_game': [
          25.0, 27.0, 29.0, 28.0,
          24.0, 25.0, 28.0, 27.0,
          33.0, 26.0, 22.0, 18.0,
          26.0, 22.0, 25.0
      ]
  })

  

• Adding new line characters when required

  See example

  Same NBA code as above.

  

• Completing docstrings

  See example

  Same NBA code as above.

  

• Don't return prefix Sometimes if you want to print out a variable and called dataframes and you start writing dat it will suggest the full completion dataframes which leaves you with the code datdataframes.

Smart Debugging

• Don't remove imports The smart debug sometimes removes imports. My guess is that because it sees that the packages are already imported, it then decides that it does not need to reimport them and removes it from the code.

• Invalid dataframe creation -- array have different lengths To fix this, it should figure out which one has the incorrect length and then correct it.

  See example dataframe

  This dataframe has one extra value in the points_per_game array

  import pandas as pd
  nba_players_df = pd.DataFrame({
      'player_name': [
          'LeBron James', 'Kevin Durant', 'Stephen Curry', 'Giannis Antetokounmpo',
          'Kawhi Leonard', 'James Harden', 'Luka Dončić', 'Damian Lillard',
          'Joel Embiid', 'Nikola Jokić', 'Anthony Davis', 'Chris Paul',
          'Jayson Tatum', 'Zion Williamson', 'Devin Booker'
      ],
      'team': [
          'Los Angeles Lakers', 'Brooklyn Nets', 'Golden State Warriors', 'Milwaukee Bucks',
          'Los Angeles Clippers', 'Brooklyn Nets', 'Dallas Mavericks', 'Portland Trail Blazers',
          'Philadelphia 76ers', 'Denver Nuggets', 'Los Angeles Lakers', 'Phoenix Suns',
          'Boston Celtics', 'New Orleans Pelicans', 'Phoenix Suns'
      ],
      'position': [
          'SF', 'PF', 'PG', 'PF',
          'SF', 'SG', 'PG', 'PG',
          'C', 'C', 'PF', 'PG',
          'SF', 'PF', 'SG'
      ],
      'points_per_game': [
          25.0, 27.0, 29.0, 28.0,
          24.0, 25.0, 28.0, 27.0,
          33.0, 26.0, 22.0, 18.0,
          26.0, 22.0, 25.0, 24.0
      ]
  })

• Sometimes the returned code is annotated with the changes. For example: slide.shapes.add_picture(chart_path, left, top, width=Inches(8)) # Corrected keyword argument. Here, we don't want the comment "# Corrected keyword argument". That can be in the chat explanation, but not the code.

• Removing comments In this example, the AI removed comments from the original code that should not have been removed.

See example dataframe

 Starting code

    from pptx import Presentation
from openpyxl import Workbook
from openpyxl.styles import Font, PatternFill, Alignment, Border, Side
from openpyxl.utils import get_column_letter
import pandas as pd
import os

def create_formatted_excel(df, excel_path):
    """
    Creates a professionally formatted Excel file from our banking data.
    This function applies corporate styling and financial data formatting.
    
    Parameters:
    df: The pandas DataFrame containing our bank analysis data
    excel_path: Where to save the formatted Excel file
    """
    # Create a new workbook and select the active sheet
    wb = Workbook()
    ws = wb.active
    ws.title = "Bank Analysis"
    
    # Write headers
    for col_idx, column in enumerate(df.columns, 1):
        cell = ws.cell(row=1, column=col_idx, value=column)
        
        # Apply header styling
        cell.font = Font(bold=True, color="FFFFFF", size=12)
        cell.fill = PatternFill(start_color="00316A", end_color="00316A", 
                              fill_type="solid")  # RBC dark blue
        cell.alignment = Alignment(horizontal="center", vertical="center")

    # Write data and apply formatting
    for row_idx, row in enumerate(df.values, 2):
        for col_idx, value in enumerate(row, 1):
            cell = ws.cell(row=row_idx, column=col_idx, value=value)
            
            # Apply specific formatting based on column type
            if df.columns[col_idx-1] == "Market Share":
                cell.number_format = "0.0%"
            elif df.columns[col_idx-1] == "Total Assets":
                cell.number_format = '"$"#,##0"B"'
                
            cell.alignment = Alignment(horizontal="center", vertical="center")
            cell.font = Font(size=11)

    # Apply borders to all cells
    border = Border(left=Side(style='thin'), right=Side(style='thin'),
                   top=Side(style='thin'), bottom=Side(style='thin'))
    
    for row in ws.iter_rows(min_row=1, max_row=len(df)+1, 
                           min_col=1, max_col=len(df.columns)):
        for cell in row:
            cell.border = border

    # Adjust column widths for readability
    for column in ws.columns:
        max_length = 0
        column_letter = get_column_letter(column[0].column)
        
        for cell in column:
            try:
                max_length = max(max_length, len(str(cell.value)))
            except:
                pass
        
        # Add some padding to the width
        adjusted_width = (max_length + 2)
        ws.column_dimensions[column_letter].width = adjusted_width

    # Save the formatted Excel file
    wb.save(excel_path)

def create_linked_powerpoint(excel_path, pptx_path):
    """
    Creates a PowerPoint presentation with a linked Excel table.
    The table will automatically update when the Excel file changes.
    
    Parameters:
    excel_path: Path to the formatted Excel file
    pptx_path: Where to save the PowerPoint file
    """
    prs = Presentation()
    slide = prs.slides.add_slide(prs.slide_layouts[1])
    
    # Add title
    title = slide.shapes.title
    title.text = "Bank Analysis Summary"
    
    # Get the content placeholder
    placeholder = slide.placeholders[1]
    
    # Create a link to the Excel file
    # We need absolute paths for the linking to work correctly
    abs_excel_path = os.path.abspath(excel_path)
    
    # Insert the Excel worksheet as a linked object
    placeholder.insert_ole_object(
        prog_id="Excel.Sheet",
        file=abs_excel_path,
        icon=False,  # Show the actual table, not an icon
        link=True    # Create a link rather than embedding
    )

# Now let's put it all together
def create_bank_analysis_presentation():
    """
    Creates both the Excel file and PowerPoint presentation with proper linking.
    This is our main function that coordinates the entire process.
    """
    # Prepare our data
    banks = pd.DataFrame({
        'stock': ['RY', 'JPM', 'BNPQY', 'TD', 'GS', 'C'],
        'name': ['Royal Bank of Canada', 'JPMorgan Chase & Co.', 'BNP Paribas', 
                'Toronto-Dominion Bank', 'Goldman Sachs', 'Citigroup'],
        'total_assets_B': [1800, 3200, 2900, 1500, 1100, 2300]
    })
    
    # Calculate market share
    banks['market_share'] = banks['total_assets_B'] / banks['total_assets_B'].sum()
    
    # Prepare presentation data
    presentation_df = banks[['name', 'stock', 'total_assets_B', 'market_share']]
    presentation_df.columns = ['Bank Name', 'Ticker', 'Total Assets', 'Market Share']
    
    # Define our file paths
    excel_path = 'bank_analysis.xlsx'
    pptx_path = 'bank_analysis.pptx'
    
    # Create the formatted Excel file
    create_formatted_excel(presentation_df, excel_path)
    
    # Create the PowerPoint with the linked table
    create_linked_powerpoint(excel_path, pptx_path)

# Run our presentation creation
create_bank_analysis_presentation()

 This error `AttributeError: 'SlidePlaceholder' object has no attribute 'insert_ole_object'`

  When it returned the code, it removed all of the comments from the original code snippet. 
</details>

Eval Infrastructure Improvements

• Move the evals to execing code in an ipython kernel
• Always have the import mitosheet code executed first

[jake-stack]

I asked Mito AI to tell me how to do simple machine learning. The first section of the response was

"Choose a Machine Learning Library: Popular libraries include scikit-learn, TensorFlow, and PyTorch. For beginners, scikit-learn is often recommended due to its simplicity and ease of use."

When I did another prompt that said "import the libraries from section 1", it imported none of these libraries and instead imported mitosheet and pandas.

It would be great if the AI could better recognize prompts that are referring to previous responses in the chat.