# Agent
Source: https://docs.pandas-ai.com/v3/agent

Build multi-turn PandasAI agents with clarifications, explanations, query rephrasing, optional sandboxed execution, and enterprise training via local vector stores.

## PandasAI Agent Overview

While the `pai.chat()` method is meant to be used in a single session and for exploratory data analysis, an agent can be used for multi-turn conversations.

To instantiate an agent, you can use the following code:

```python
import os
from pandasai import Agent
import pandas as pd

# Sample DataFrames
sales_by_country = pd.DataFrame({
    "country": ["United States", "United Kingdom", "France", "Germany", "Italy", "Spain", "Canada", "Australia", "Japan", "China"],
    "sales": [5000, 3200, 2900, 4100, 2300, 2100, 2500, 2600, 4500, 7000],
    "deals_opened": [142, 80, 70, 90, 60, 50, 40, 30, 110, 120],
    "deals_closed": [120, 70, 60, 80, 50, 40, 30, 20, 100, 110]
})

agent = Agent(sales_by_country)
agent.chat('Which are the top 5 countries by sales?')
# Output: China, United States, Japan, Germany, Australia
```

Contrary to the `pai.chat()` method, an agent will keep track of the state of the conversation and will be able to answer multi-turn conversations. For example:

```python
agent.chat('And which one has the most deals?')
# Output: United States has the most deals
```

### Clarification questions

An agent will also be able to ask clarification questions if it does not have enough information to answer the query. For example:

```python
agent.clarification_questions('What is the GDP of the United States?')
```

This will return up to 3 clarification questions that the agent can ask the user to get more information to answer the query.

### Explanation

An agent will also be able to explain the answer given to the user. For example:

```python
response = agent.chat('What is the GDP of the United States?')
explanation = agent.explain()

print("The answer is", response)
print("The explanation is", explanation)
```

### Rephrase Question

Rephrase question to get accurate and comprehensive response from the model. For example:

```python
rephrased_query = agent.rephrase_query('What is the GDP of the United States?')

print("The rephrased query is", rephrased_query)

```

## Using the Agent in a Sandbox Environment

<Note>
  The sandbox works offline and provides an additional layer of security for
  code execution. It's particularly useful when working with untrusted data or
  when you need to ensure that code execution is isolated from your main system.
</Note>

To enhance security and protect against malicious code through prompt injection, PandasAI provides a sandbox environment for code execution. The sandbox runs your code in an isolated Docker container, ensuring that potentially harmful operations are contained.

### Installation

Before using the sandbox, you need to install Docker on your machine and ensure it is running.

First, install the sandbox package:

```bash
pip install pandasai-docker
```

### Basic Usage

Here's how to use the sandbox with your PandasAI agent:

```python
from pandasai import Agent
from pandasai_docker import DockerSandbox

# Initialize the sandbox
sandbox = DockerSandbox()
sandbox.start()

# Create an agent with the sandbox
df = pai.read_csv("data.csv")
agent = Agent([df], sandbox=sandbox)

# Chat with the agent - code will run in the sandbox
response = agent.chat("Calculate the average sales")

# Don't forget to stop the sandbox when done
sandbox.stop()
```

### Customizing the Sandbox

You can customize the sandbox environment by specifying a custom name and Dockerfile:

```python
sandbox = DockerSandbox(
    "custom-sandbox-name",
    "/path/to/custom/Dockerfile"
)
```

## Training the Agent with local Vector stores

<Note>
  Training agents with local vector stores requires a PandasAI Enterprise license. See [Enterprise Features](/v3/enterprise-features) for more details or [contact us](https://pandas-ai.com/) for production use.
</Note>

It is possible also to use PandasAI with a few-shot learning agent, thanks to the "train with local vector store" enterprise feature (requiring an enterprise license).

If you want to train the agent with a local vector store, you can use the local `ChromaDB`, `Qdrant` or `Pinecone` vector stores. Here's how to do it:
An enterprise license is required for using the vector stores locally. See [Enterprise Features](/v3/enterprise-features) for licensing information.
If you plan to use it in production, [contact us](https://pandas-ai.com).

```python
from pandasai import Agent
from pandasai.ee.vectorstores import ChromaDB
from pandasai.ee.vectorstores import Qdrant
from pandasai.ee.vectorstores import Pinecone
from pandasai.ee.vector_stores import LanceDB

# Instantiate the vector store
vector_store = ChromaDB()
# or with Qdrant
# vector_store = Qdrant()
# or with LanceDB
vector_store = LanceDB()
# or with Pinecone
# vector_store = Pinecone(
#     api_key="*****",
#     embedding_function=embedding_function,
#     dimensions=384, # dimension of your embedding model
# )

# Instantiate the agent with the custom vector store
agent = Agent("data.csv", vectorstore=vector_store)

# Train the model
query = "What is the total sales for the current fiscal year?"
# The following code is passed as a string to the response variable
response = '\n'.join([
    'import pandas as pd',
    '',
    'df = dfs[0]',
    '',
    '# Calculate the total sales for the current fiscal year',
    'total_sales = df[df[\'date\'] >= pd.to_datetime(\'today\').replace(month=4, day=1)][\'sales\'].sum()',
    'result = { "type": "number", "value": total_sales }'
])

agent.train(queries=[query], codes=[response])

response = agent.chat("What is the total sales for the last fiscal year?")
print(response)
# The model will use the information provided in the training to generate a response
```


# Chat and Output Formats
Source: https://docs.pandas-ai.com/v3/chat-and-output

Learn how to use PandasAI's powerful chat functionality and the output formats for natural language data analysis

## Chat

The `.chat()` method is PandasAI's core feature that enables natural language interaction with your data. It allows you to:

* Query your data using plain English
* Generate visualizations and statistical analyses
* Work with multiple DataFrames simultaneously

### Basic Usage

```python
import pandasai as pai

df_customers = pai.read_csv("customers.csv")

response = df_customers.chat("Which are our top 5 customers?")
```

### Chat with multiple DataFrames

```python
import pandasai as pai

df_customers = pai.read_csv("customers.csv")
df_orders = pai.read_csv("orders.csv")
df_products = pai.read_csv("products.csv")

response = pai.chat('Who are our top 5 customers and what products do they buy most frequently?', df_customers, df_orders, df_products)
```

## Available Output Formats

PandasAI supports multiple output formats for responses, each designed to handle different types of data and analysis results effectively. This document outlines the available output formats and their use cases.

### DataFrame Response

Used when the result is a pandas DataFrame. This format preserves the tabular structure of your data and allows for further data manipulation.

### Chart Response

Handles visualization outputs, supporting various types of charts and plots generated during data analysis.

### String Response

Returns textual responses, explanations, and insights about your data in a readable format.

### Number Response

Specialized format for numerical outputs, typically used for calculations, statistics, and metrics.

### Error Response

Provides structured error information when something goes wrong during the analysis process.

## Usage

The response format is automatically determined based on the type of analysis performed and the nature of the output. You don't need to explicitly specify the format - PandasAI will choose the most appropriate one for your results.

Example:

```python
import pandasai as pai

df = pai.read_csv("users.csv")

response = df.chat("Who is the user with the highest age?") # Returns a String response
response = df.chat("How many users in total?") # Returns a Number response
response = df.chat("Show me the data") # Returns a DataFrame response
response = df.chat("Plot the distribution") # Returns a Chart response
```

## Response Types Details

Each response type is designed to handle specific use cases:

* **String Response**: Provides textual analysis and explanations
* **Number Response**: Returns numerical results from calculations
* **DataFrame Response**: Preserves the structure and functionality of pandas DataFrames
* **Chart Response**: Handles various visualization formats and plotting libraries
* **Error Response**: Structured error handling with informative messages

The response system is extensible and type-safe, ensuring that outputs are properly formatted and handled according to their specific requirements.

## Response Object Methods

The response object provides several useful methods and properties to interact with the results:

### Value Property

By default, when you print a response object, it automatically returns its `.value` property:

```python
response = df.chat("What is the average age?")
print(response)  # Automatically calls response.value
# Output: The average age is 34.5 years

# For charts, printing will display the visualization
chart_response = df.chat("Plot age distribution")
print(chart_response)  # Displays the chart
```

### Generated Code

You can inspect the code that was generated to produce the result:

```python
response = df.chat("Calculate the correlation between age and salary")
print(response.last_code_executed)
# Output: df['age'].corr(df['salary'])
```

### Saving Charts

For chart responses, you can save the visualization to a file:

```python
chart_response = df.chat("Create a scatter plot of age vs salary")
chart_response.save("scatter_plot.png")  # Saves the chart as PNG
```


# null
Source: https://docs.pandas-ai.com/v3/contributing



# 🐼 Contributing to PandasAI

Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great.

## 🤝 How to submit a contribution

To make a contribution, follow the following steps:

1. Fork and clone this repository
2. Do the changes on your fork
3. If you modified the code (new feature or bug-fix), please add tests for it
4. Check the linting [see below](#linting)
5. Ensure that all tests pass [see below](#testing)
6. Submit a pull request

For more details about pull requests, please read [GitHub's guides](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request).

### 📦 Package manager

We use `poetry` as our package manager. You can install poetry by following the instructions [here](https://python-poetry.org/docs/#installation).

Please DO NOT use pip or conda to install the dependencies. Instead, use poetry:

```bash
poetry install --all-extras --with dev
```

### 📌 Pre-commit

To ensure our standards, make sure to install pre-commit before starting to contribute.

```bash
pre-commit install
```

### 🧹 Linting

We use `ruff` to lint our code. You can run the linter by running the following command:

```bash
make format_diff
```

Make sure that the linter does not report any errors or warnings before submitting a pull request.

### Code Format with `ruff-format`

We use `ruff` to reformat the code by running the following command:

```bash
make format
```

### Spell check

We usee `codespell` to check the spelling of our code. You can run codespell by running the following command:

```bash
make spell_fix
```

### 🧪 Testing

We use `pytest` to test our code. You can run the tests by running the following command:

```bash
make test_all
```

Make sure that all tests pass before submitting a pull request.

## 🚀 Release Process

At the moment, the release process is manual. We try to make frequent releases. Usually, we release a new version when we have a new feature or bugfix. A developer with admin rights to the repository will create a new release on GitHub, and then publish the new version to PyPI.


# Enterprise License
Source: https://docs.pandas-ai.com/v3/enterprise-features

Features requiring PandasAI Enterprise license

## License Information

Code under the `ee/` folder requires a PandasAI Enterprise license for production use. Everything else is under MIT license.

For licensing inquiries, visit [pandas-ai.com](https://pandas-ai.com/).

## Enterprise Features & Connectors

<table style={{ borderCollapse: 'collapse', width: '100%', border: '1px solid #ccc' }}>
  <tr>
    <th style={{ border: '1px solid #ccc', padding: '8px 16px', textAlign: 'left' }}>Feature/Connector</th>
    <th style={{ border: '1px solid #ccc', padding: '8px 16px', textAlign: 'left' }}>Type</th>
    <th style={{ border: '1px solid #ccc', padding: '8px 16px', textAlign: 'left' }}>Extension</th>
    <th style={{ border: '1px solid #ccc', padding: '8px 16px', textAlign: 'left' }}>Documentation</th>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Snowflake</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Connector</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai-snowflake</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><a href="/v3/semantic-layer/data-ingestion#snowflake-extension-ee">Snowflake Docs</a></td>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Databricks</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Connector</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai-databricks</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><a href="/v3/semantic-layer/data-ingestion#databricks-extension-ee">Databricks Docs</a></td>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>BigQuery</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Connector</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai-bigquery</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><a href="/v3/semantic-layer/data-ingestion#bigquery-extension-ee">BigQuery Docs</a></td>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Oracle</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Connector</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai-oracle</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><a href="/v3/semantic-layer/data-ingestion#oracle-extension-ee">Oracle Docs</a></td>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Skills</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Feature</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai (ee)</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><a href="/v3/skills">Skills</a></td>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Vector Stores (Training)</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Feature</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai (ee)</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><a href="/v3/agent#training-the-agent-with-local-vector-stores">Agent Training</a></td>
  </tr>
</table>


# Installation & Quickstart
Source: https://docs.pandas-ai.com/v3/getting-started

Start building your data preparation layer with PandasAI and chat with your data

## Installation

PandasAI requires Python `3.8+ <=3.11`. We recommend using Poetry for dependency management:

```bash
# Using poetry (recommended)
poetry add pandasai

# Alternative: using pip
pip install pandasai
```

## Quick setup

In order to use PandasAI, you need a large language model (LLM). You can use any LLM, but for this guide we'll use OpenAI through the LiteLLM extension.

First, install the required extension:

```bash
pip install pandasai-litellm
```

Then, import PandasAI and configure the LLM:

```python
import pandasai as pai
from pandasai_litellm.litellm import LiteLLM

# Initialize LiteLLM with your OpenAI model
llm = LiteLLM(model="gpt-4.1-mini", api_key="YOUR_OPENAI_API_KEY")

# Configure PandasAI to use this LLM
pai.config.set({
    "llm": llm
})
```

## Chat with your data

```python
import pandasai as pai
from pandasai_litellm.litellm import LiteLLM

# Initialize LiteLLM with your OpenAI model
llm = LiteLLM(model="gpt-4.1-mini", api_key="YOUR_OPENAI_API_KEY")

# Configure PandasAI to use this LLM
pai.config.set({
    "llm": llm
})

# Load your data
df = pai.read_csv("data/companies.csv")

response = df.chat("What is the average revenue by region?")
print(response)
```

When you ask a question, PandasAI will use the LLM to generate the answer and output a response.
Depending on your question, it can return different kind of responses:

* string
* dataframe
* chart
* number

Find it more about output data formats [here](/v3/chat-and-output#available-output-formats).

## Next Steps

* [Config NL Layer](/v3/overview-nl)
* [Set up LLM](/v3/large-language-models)


# Introduction to PandasAI
Source: https://docs.pandas-ai.com/v3/introduction

PandasAI is a Python library that makes it easy to ask questions to your data in natural language.

# ![PandasAI](https://github.com/Sinaptik-AI/pandas-ai/blob/main/assets/logo.png?raw=true)

Beyond querying, PandasAI offers functionalities to visualize data through graphs, cleanse datasets by addressing missing values, and enhance data quality through feature generation, making it a comprehensive tool for data scientists and analysts.

## Features

* **Natural language querying**: Ask questions to your data in natural language.
* **Data visualization**: Generate graphs and charts to visualize your data.
* **Data cleansing**: Cleanse datasets by addressing missing values.
* **Feature generation**: Enhance data quality through feature generation.
* **Data connectors**: Connect to various data sources like CSV, XLSX, PostgreSQL, MySQL, BigQuery, Databricks, Snowflake, etc.

## How does PandasAI work?

PandasAI uses generative AI models to understand and interpret natural language queries and translate them into python code and SQL queries. It then uses the code to interact with the data and return the results to the user.

## Who should use PandasAI?

PandasAI is designed for business analysts, data scientists, and engineers who want to interact with their data in a more natural way. It is particularly useful for those who are not familiar with SQL or Python or who want to save time and effort when working with data. It is also useful for those who are familiar with SQL and Python, as it allows them to ask questions to their data without having to write any complex code.

## How to get started with PandasAI?

PandasAI is available as a Python library. You can install the library using pip or poetry and use it in your Python code.

### 📚 Using the library

The PandasAI library provides a Python interface for interacting with your data in natural language. You can use it to ask questions to your data, generate graphs and charts, cleanse datasets, and enhance data quality through feature generation. It uses LLMs to understand and interpret natural language queries and translate them into python code and SQL queries.

Once you have installed pandasai, simply import it and use it to ask questions to your data.

```python
import pandasai as pai
from pandasai_litellm.litellm import LiteLLM

# Initialize LiteLLM with your OpenAI model
llm = LiteLLM(model="gpt-4.1-mini", api_key="YOUR_OPENAI_API_KEY")

# Configure PandasAI to use this LLM
pai.config.set({
    "llm": llm
})

# Load your data
df = pai.read_csv("data/companies.csv")

response = df.chat("What is the average revenue by region?")
print(response)
```

## Support

If you have any questions or need help, please join our **[discord server](https://discord.gg/KYKj9F2FRH)**.

## License

PandasAI is available under the MIT expat license, except for the `pandasai/ee` directory, which has its [license here](https://github.com/Sinaptik-AI/pandas-ai/blob/master/pandasai/ee/LICENSE) if applicable.

If you are interested in the Enterprise License, see [Enterprise Features](/v3/enterprise-features) or visit [pandas-ai.com](https://pandas-ai.com/).

## Analytics

We've partnered with [Scarf](https://scarf.sh) to collect anonymized user statistics to understand which features our community is using and how to prioritize product decision-making in the future. To opt out of this data collection, you can set the environment variable `SCARF_NO_ANALYTICS=true`.


# Set up LLM
Source: https://docs.pandas-ai.com/v3/large-language-models

Set up Large Language Model in PandasAI

PandasAI supports multiple LLMs.
You need to install the corresponding LLM extension.
Once an LLM extension is installed, you can configure it using [`pai.config.set()`](/v3/overview-nl#configure-the-nl-layer).
Then, every time you use the [`.chat()`](/v3/chat-and-output) method, it will use the configured LLM.

## LiteLLM

LiteLLM provides a unified interface to multiple LLM providers including OpenAI, Anthropic, Google, and others.

Install the pandasai-litellm extension:

```bash
pip install pandasai-litellm
```

Then configure it in your code:

```python
import pandasai as pai
from pandasai_litellm.litellm import LiteLLM

# For OpenAI models
llm = LiteLLM(model="gpt-4.1-mini", api_key="YOUR_OPENAI_API_KEY")

# For other providers, change the model name and provide appropriate credentials
# llm = LiteLLM(model="anthropic/claude-3-opus-20240229", api_key="YOUR_ANTHROPIC_API_KEY")

pai.config.set({
    "llm": llm
})
```

## OpenAI models

Install the pandasai-openai extension:

```bash
# Using poetry
poetry add pandasai-openai

# Using pip
pip install pandasai-openai
```

In order to use OpenAI models, you need to have an OpenAI API key. You can get one here.
Once you have an API key, you can use it to instantiate an OpenAI object:

Configure OpenAI:

```python
import pandasai as pai
from pandasai_openai import OpenAI

llm = OpenAI(api_token="my-openai-api-key")

# Set your OpenAI API key
pai.config.set({"llm": llm})
```

### Azure OpenAI models

Install the pandasai-openai extension:

```bash
# Using poetry
poetry add pandasai-openai

# Using pip
pip install pandasai-openai
```

In order to use Azure OpenAI models, you need to have an Azure OpenAI API key. You can get one here.
Once you have an API key, you can use it to instantiate an Azure OpenAI object:

Configure Azure OpenAI:

```python
import pandasai as pai
from pandasai_openai import AzureOpenAI

llm = AzureOpenAI(api_base="https://<your-endpoint>.openai.azure.com/",
    api_key="my-azure-openai-api-key",
    deployment_name="text-davinci-003")  # The name of your deployed model

pai.config.set({"llm": llm})
```

## How to set up any LLM?

LiteLLM provides a unified interface to interact with 100+ LLM models from various providers including OpenAI, Azure, Anthropic, Google, AWS, Hugging Face, and many more. This makes it easy to switch between different LLM providers without changing your code.

Install the pandasai-litellm extension:

```bash
# Using poetry
poetry add pandasai-litellm

# Using pip
pip install pandasai-litellm
```

Configure LiteLLM with your chosen model. First, set up your API keys as environment variables:

```python
import os
import pandasai as pai
from pandasai_litellm import LiteLLM

# Set your API keys as environment variables
os.environ["OPENAI_API_KEY"] = "your-openai-api-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-api-key"

# Example with OpenAI
llm = LiteLLM(model="gpt-4.1-mini")

# Example with Anthropic
llm = LiteLLM(model="claude-2")

# Set your LLM configuration
pai.config.set({"llm": llm})
```

LiteLLM supports a wide range of models from various providers, including but not limited to:

* OpenAI (gpt-4.1-mini, gpt-4, etc.)
* Anthropic (claude-2, claude-instant-1, etc.)
* Google (gemini-pro, palm2, etc.)
* Azure OpenAI
* AWS (Bedrock, SageMaker)
* Mistral AI
* Cohere
* Hugging Face

For a complete list of supported models and providers, visit the [LiteLLM documentation](https://docs.litellm.ai/docs/providers).

## Determinism

Determinism in language models refers to the ability to produce the same output consistently given the same input under identical conditions. This characteristic is vital for:

* Reproducibility: Ensuring the same results can be obtained across different runs, which is crucial for debugging and iterative development.
* Consistency: Maintaining uniformity in responses, particularly important in scenarios like automated customer support, where varied responses to the same query might be undesirable.
* Testing: Facilitating the evaluation and comparison of models or algorithms by providing a stable ground for testing.

### The Role of temperature=0

The temperature parameter in language models controls the randomness of the output. A higher temperature increases diversity and creativity in responses, while a lower temperature makes the model more predictable and conservative. Setting `temperature=0` essentially turns off randomness, leading the model to choose the most likely next word at each step. This is critical for achieving determinism as it minimizes variance in the model's output.

### Implications of temperature=0

* Predictable Responses: The model will consistently choose the most probable path, leading to high predictability in outputs.
* Creativity: The trade-off for predictability is reduced creativity and variation in responses, as the model won't explore less likely options.

### Utilizing seed for Enhanced Control

The seed parameter is another tool to enhance determinism. It sets the initial state for the random number generator used in the model, ensuring that the same sequence of "random" numbers is used for each run. This parameter, when combined with `temperature=0`, offers an even higher degree of predictability.

### Example:

```python
import pandasai as pai

# Sample DataFrame
df = pai.DataFrame({
    "country": ["United States", "United Kingdom", "France", "Germany", "Italy", "Spain", "Canada", "Australia", "Japan", "China"],
    "gdp": [19294482071552, 2891615567872, 2411255037952, 3435817336832, 1745433788416, 1181205135360, 1607402389504, 1490967855104, 4380756541440, 14631844184064],
    "happiness_index": [6.94, 7.16, 6.66, 7.07, 6.38, 6.4, 7.23, 7.22, 5.87, 5.12]
})

# Configure the LLM
pai.config.set("temperature", 0)
pai.config.set("seed", 26)

df.chat('Which are the 5 happiest countries?') # answer should me (mostly) consistent across devices.
```

### Current Limitation:

#### AzureOpenAI Instance

While the seed parameter is effective with the OpenAI instance in our library, it's important to note that this functionality is not yet available for AzureOpenAI. Users working with AzureOpenAI can still use `temperature=0` to reduce randomness but without the added predictability that seed offers.

#### System fingerprint

As mentioned in the documentation ([OpenAI Seed](https://platform.openai.com/docs/guides/text-generation/reproducible-outputs)) :

> Sometimes, determinism may be impacted due to necessary changes OpenAI makes to model configurations on our end. To help you keep track of these changes, we expose the system\_fingerprint field. If this value is different, you may see different outputs due to changes we've made on our systems.

### Workarounds and Future Updates

For AzureOpenAI Users: Rely on `temperature=0` for reducing randomness. Stay tuned for future updates as we work towards integrating seed functionality with AzureOpenAI.
For OpenAI Users: Utilize both `temperature=0` and seed for maximum determinism.


# null
Source: https://docs.pandas-ai.com/v3/license



Copyright (c) 2023 Sinaptik GmbH

Portions of this software are licensed as follows:

* All content that resides under any "pandasai/ee/" directory of this repository, if such directories exists, are licensed under the license defined in "pandasai/ee/LICENSE".
* All third party components incorporated into the PandasAI Software are licensed under the original license provided by the owner of the applicable component.
* Content outside of the above mentioned directories or restrictions above is available under the "MIT Expat" license as defined below.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.


# Backwards Compatibility
Source: https://docs.pandas-ai.com/v3/migration-backwards-compatibility

Using v2 classes in PandasAI v3

<Note>
  PandasAI v3 maintains backward compatibility for `SmartDataframe`, `SmartDatalake`, and `Agent`. However, we recommend migrating to the new `pai.DataFrame()` and `pai.chat()` methods for better performance and features.
</Note>

## SmartDataframe

`SmartDataframe` continues to work in v3 with the same API. However, you must configure the LLM globally.

### Using SmartDataframe in v3 (Legacy)

```python
from pandasai import SmartDataframe
import pandasai as pai
import pandas as pd
from pandasai_litellm.litellm import LiteLLM

# Configure LLM globally (required)
llm = LiteLLM(model="gpt-4o-mini", api_key="your-api-key")
pai.config.set({"llm": llm})

# v2 style still works
df = pd.DataFrame({
    "country": ["US", "UK", "France"],
    "sales": [5000, 3200, 2900]
})

smart_df = SmartDataframe(df)
response = smart_df.chat("What are the top countries by sales?")
```

### Recommended v3 Approach

While `SmartDataframe` works, we recommend using `pai.DataFrame()` for better integration with v3 features:

```python
import pandasai as pai
import pandas as pd

# Configure LLM globally
pai.config.set({"llm": llm})

# Simple approach
df = pd.DataFrame({
    "country": ["US", "UK", "France"],
    "sales": [5000, 3200, 2900]
})
df = pai.DataFrame(df)
response = df.chat("What are the top countries by sales?")
```

**Benefits of pai.DataFrame():**

* Better integration with semantic layer
* Improved context management
* Enhanced performance
* Access to v3-specific features
* Cleaner API

## SmartDatalake

`SmartDatalake` still works but is no longer necessary. You can query multiple dataframes directly with `pai.chat()`.

### Using SmartDatalake in v3 (Legacy)

```python
from pandasai import SmartDatalake
import pandasai as pai
import pandas as pd
from pandasai_litellm.litellm import LiteLLM

# Configure LLM globally (required)
llm = LiteLLM(model="gpt-4o-mini", api_key="your-api-key")
pai.config.set({"llm": llm})

# v2 style still works
employees_df = pd.DataFrame({
    "name": ["John", "Jane", "Bob"],
    "department": ["Sales", "Engineering", "Sales"]
})

salaries_df = pd.DataFrame({
    "name": ["John", "Jane", "Bob"],
    "salary": [60000, 80000, 55000]
})

lake = SmartDatalake([
    employees_df,
    salaries_df
])

response = lake.chat("Who gets paid the most?")
```

### Recommended v3 Approach

Query multiple dataframes directly without `SmartDatalake`:

```python
import pandasai as pai

# Configure LLM globally
pai.config.set({"llm": llm})

# Create dataframes
employees = pai.DataFrame(employees_df)
salaries = pai.DataFrame(salaries_df)

# Query across multiple dataframes directly
response = pai.chat("Who gets paid the most?", employees, salaries)
```

**Benefits of pai.chat():**

* No need to instantiate `SmartDatalake`
* Cleaner, more intuitive API
* Better performance
* Semantic layer support
* Easier to add/remove dataframes dynamically

## Agent

The `Agent` class works the same way in v3 as it did in v2. The only requirement is to configure the LLM globally.

```python
from pandasai import Agent
import pandasai as pai
from pandasai_litellm.litellm import LiteLLM

# Configure LLM globally (required in v3)
llm = LiteLLM(model="gpt-4o-mini", api_key="your-api-key")
pai.config.set({"llm": llm})

# Agent works as before
df1 = pai.DataFrame(sales_data)
df2 = pai.DataFrame(costs_data)

agent = Agent([df1, df2])
response = agent.chat("Analyze the data and provide insights")
```

**Key Change:** Configure LLM globally with `pai.config.set()` instead of passing it per-agent.

For detailed information about Agent usage, see the [Agent documentation](/v3/agent). For information about using Skills with Agent, see the [Skills documentation](/v3/skills).


# Migration Guide: PandasAI v2 to v3
Source: https://docs.pandas-ai.com/v3/migration-guide

Step-by-step guide to migrate from PandasAI v2 to v3

<Note title="Migration Notice">
  PandasAI 3.0 introduces significant architectural changes. This guide covers
  breaking changes and migration steps. See [Backwards
  Compatibility](/v3/migration-backwards-compatibility) for v2 classes that
  still work.
</Note>

## Breaking Changes

### Configuration

Configuration is now global using `pai.config.set()` instead of per-dataframe. Several options have been removed:

**Removed:** `save_charts`, `enable_cache`, `security`, `custom_whitelisted_dependencies`, `save_charts_path`, `custom_head`

**v2:**

```python
from pandasai import SmartDataframe

config = {
    "llm": llm,
    "save_charts": True,
    "enable_cache": True,
    "security": "standard"
}
df = SmartDataframe(data, config=config)
```

**v3:**

```python
import pandasai as pai

pai.config.set({
    "llm": llm,
    "save_logs": True,
    "verbose": False,
    "max_retries": 3
})
df = pai.DataFrame(data)
```

**Key Changes:**

* Global configuration applies to all dataframes
* Charts returned as `ChartResponse` objects for manual handling
* Security handled through sandbox environment
* Caching removed for simplicity

**More details:** See [config docs](/v3/overview-nl#configure-the-nl-layer) for configuration examples and more details.

### LLM

LLMs are now extension-based. Install `pandasai-litellm` separately for unified access to 100+ models.

**v2:**

```python
from pandasai.llm import OpenAI
from pandasai import SmartDataframe

llm = OpenAI(api_token="your-api-key")
df = SmartDataframe(data, config={"llm": llm})
```

**v3:**

```bash
pip install pandasai-litellm
```

```python
import pandasai as pai
from pandasai_litellm.litellm import LiteLLM

llm = LiteLLM(model="gpt-4o-mini", api_key="your-api-key")
pai.config.set({"llm": llm})
df = pai.DataFrame(data)
```

**Key Changes:**

* LLMs are now extension-based, not built-in
* Install `pandasai-litellm` for unified LLM interface
* LiteLLM supports 100+ models (GPT-4, Claude, Gemini, etc.)
* Configure LLM globally instead of per-dataframe
* You need to install both `pandasai` and `pandasai-litellm`

**More details:** See [Large Language Models](/v3/large-language-models) for supported models and configuration.

### Data Connectors

Connectors are now separate extensions. Install only what you need. Cloud connectors require [enterprise license](/v3/enterprise-features).

**v2:**

```python
from pandasai.connectors import PostgreSQLConnector
from pandasai import SmartDataframe

connector = PostgreSQLConnector(config={
    "host": "localhost",
    "database": "mydb",
    "table": "sales"
})
df = SmartDataframe(connector)
```

**v3:**

```bash
pip install pandasai-sql[postgres]
```

```python
import pandasai as pai

df = pai.create(
    path="company/sales",
    description="Sales data from PostgreSQL",
    source={
        "type": "postgres",
        "connection": {
            "host": "localhost",
            "database": "mydb",
            "user": "${DB_USER}",
            "password": "${DB_PASSWORD}"
        },
        "table": "sales"
    }
)
```

**Key Changes:**

* Install specific extensions: `pandasai-sql[postgres]`, `pandasai-sql[mysql]`
* Use `pai.create()` with semantic layer
* Environment variables supported: `${DB_USER}`

**More details:** See [Data Ingestion](/v3/semantic-layer/data-ingestion) for connector setup and configuration.

### Skills

<Note title="Enterprise Feature">
  Skills require a valid enterprise license for production use. See [Enterprise
  Features](/v3/enterprise-features) for more details.
</Note>

Skills use `@pai.skill` decorator and are automatically registered globally.

**v2:**

```python
from pandasai.skills import skill
from pandasai import Agent

@skill
def calculate_bonus(salary: float, performance: float) -> float:
    """Calculate employee bonus."""
    if performance >= 90:
        return salary * 0.15
    return salary * 0.10

agent = Agent([df])
agent.add_skills(calculate_bonus)
```

**v3:**

```python
import pandasai as pai
from pandasai import Agent

@pai.skill
def calculate_bonus(salary: float, performance: float) -> float:
    """Calculate employee bonus."""
    if performance >= 90:
        return salary * 0.15
    return salary * 0.10

# Skills automatically available - no need to add them
agent = Agent([df])
```

**Key Changes:**

* Use `@pai.skill` instead of `@skill`
* Automatic global registration
* No need for `agent.add_skills()`
* Works with `pai.chat()`, `SmartDataframe`, and `Agent`

**More details:** See [Skills](/v3/skills) for detailed usage and examples.

### Training

<Note title="Enterprise Feature">
  Training with vector stores requires a valid enterprise license for production
  use. See [Enterprise Features](/v3/enterprise-features) for more details.
</Note>

Training is now available through local vector stores (ChromaDB, Qdrant, Pinecone, LanceDB) for few-shot learning. The `train()` method is still available but requires a vector store.

**v2:**

```python
from pandasai import Agent

agent = Agent(df)
agent.train(queries=["query"], codes=["code"])
```

**v3:**

```python
from pandasai import Agent
from pandasai.ee.vectorstores import ChromaDB

# Instantiate with vector store
vector_store = ChromaDB()
agent = Agent(df, vectorstore=vector_store)

# Train with vector store
agent.train(queries=["query"], codes=["code"])
```

**Key Changes:**

* Training requires a vector store (ChromaDB, Qdrant, Pinecone, LanceDB)
* Vector stores enable few-shot learning
* Better scalability and performance

**More details:** See [Training the Agent](/v3/agent#training-the-agent-with-local-vector-stores) for setup and examples.

## Migration Steps

### Step 1: Update Installation

```bash
# Using pip
pip install pandasai pandasai-litellm

# Using poetry
poetry add pandasai pandasai-litellm

# For SQL connectors
pip install pandasai-sql[postgres]  # or mysql, sqlite, etc.
```

### Step 2: Update Imports

```python
# v2 imports
from pandasai import SmartDataframe, SmartDatalake, Agent
from pandasai.llm import OpenAI
from pandasai.skills import skill
from pandasai.connectors import PostgreSQLConnector

# v3 imports
import pandasai as pai
from pandasai import Agent
from pandasai_litellm.litellm import LiteLLM
```

### Step 3: Configure LLM Globally

```python
from pandasai_litellm.litellm import LiteLLM
import pandasai as pai

llm = LiteLLM(model="gpt-4o-mini", api_key="your-api-key")
pai.config.set({
    "llm": llm,
    "verbose": False,
    "save_logs": True,
    "max_retries": 3
})
```

### Step 4: Migrate DataFrames (optional)

Check the [Backwards Compatibility](/v3/migration-backwards-compatibility) section for details on the difference between SmartDataframe, SmartDatalakes, and the new Semantic DataFrames (pai dataframes).
In this way you can decide if migrating or not.

**Option A: Keep SmartDataframe (backward compatible)**

```python
from pandasai import SmartDataframe

df = SmartDataframe(your_data)
response = df.chat("Your question")
```

**Option B: Use pai.DataFrame (recommended)**

```python
import pandasai as pai

# Simple approach
df = pai.DataFrame(your_data)
response = df.chat("Your question")

# With semantic layer (best for production)
df = pai.create(
    path="company/sales-data",
    df=your_data,
    description="Sales data by country and region",
    columns={
        "country": {"type": "string", "description": "Country name"},
        "sales": {"type": "float", "description": "Sales amount in USD"}
    }
)
response = df.chat("Your question")
```

**Multiple DataFrames:**

```python
# v2 style (still works)
from pandasai import SmartDatalake
lake = SmartDatalake([df1, df2])

# v3 recommended
import pandasai as pai
df1 = pai.DataFrame(data1)
df2 = pai.DataFrame(data2)
response = pai.chat("Your question", df1, df2)
```

### Step 5: Migrate Data Connectors

```python
# v2
from pandasai.connectors import PostgreSQLConnector
connector = PostgreSQLConnector(config={...})
df = SmartDataframe(connector)

# v3
import pandasai as pai
df = pai.create(
    path="company/database-table",
    description="Description of your data",
    source={
        "type": "postgres",
        "connection": {
            "host": "localhost",
            "database": "mydb",
            "user": "${DB_USER}",
            "password": "${DB_PASSWORD}"
        },
        "table": "your_table"
    }
)
```

### Step 6: Update Skills (if applicable)

<Note title="Enterprise Feature">
  Skills require a valid enterprise license for production use. See [Enterprise
  Features](/v3/enterprise-features) for more details.
</Note>

```python
# v2
from pandasai.skills import skill
@skill
def calculate_metric(value: float) -> float:
    """Calculate custom metric."""
    return value * 1.5
agent.add_skills(calculate_metric)

# v3
import pandasai as pai
@pai.skill
def calculate_metric(value: float) -> float:
    """Calculate custom metric."""
    return value * 1.5
# Skills automatically available
```

### Step 7: Remove Deprecated Configuration

```python
# Remove: save_charts, enable_cache, security,
# custom_whitelisted_dependencies, save_charts_path

# v3 (keep only these)
pai.config.set({
    "llm": llm,
    "save_logs": True,
    "verbose": False,
    "max_retries": 3
})
```

## Migration Tests

Test your migration with these examples:

### Basic Chat Test

```python
import pandasai as pai
import pandas as pd

df = pd.DataFrame({"x": [1, 2, 3], "y": [4, 5, 6]})
df = pai.DataFrame(df)
response = df.chat("What is the sum of x?")
print(response)
```

### Multi-DataFrame Test

```python
df1 = pai.DataFrame({"sales": [100, 200, 300]})
df2 = pai.DataFrame({"costs": [50, 100, 150]})
response = pai.chat("What is the total profit?", df1, df2)
print(response)
```

### Skills Test

```python
@pai.skill
def test_skill(x: int) -> int:
    """Double the value."""
    return x * 2

df = pai.DataFrame({"values": [1, 2, 3]})
response = df.chat("Double the first value")
print(response)
```

***

<Note>
  **Next Steps:** - Review [Backwards
  Compatibility](/v3/migration-backwards-compatibility) for v2 classes - Check
  [Migration Troubleshooting](/v3/migration-troubleshooting) for common issues
</Note>


# Migration Troubleshooting
Source: https://docs.pandas-ai.com/v3/migration-troubleshooting

Common issues and solutions when migrating from v2 to v3

<Note>
  This guide covers common issues encountered during migration. For breaking changes and migration steps, see the [Migration Guide](/v3/migration-guide).
</Note>

## Common Issues and Solutions

### Issue: LLM Not Found

**Problem**: `ModuleNotFoundError: No module named 'pandasai.llm'`

**Solution**: Install the appropriate LLM extension

```bash
pip install pandasai-litellm
```

### Issue: Skills Not Working

**Problem**: Skills not being recognized

**Solution**: Use the new `@pai.skill()` decorator

```python
# v2
from pandasai.skills import skill
@skill
def my_skill():
    pass

# v3
import pandasai as pai
@pai.skill()
def my_skill():
    "doc string"
    pass
```

### Issue: Configuration Not Applied

**Problem**: Configuration settings not taking effect

**Solution**: Use global configuration

```python
# v2
df = SmartDataframe(data, config=config)

# v3
pai.config.set(config)
df = pai.DataFrame(data)
```

## Get Support

### Community Support

If you need help with migration or have questions, join our **[Discord community](https://discord.gg/KYKj9F2FRH)** where you can get support from other PandasAI users and contributors.

### Enterprise Support

Enterprise customers should contact their dedicated account manager via Slack or through the dedicated support channel selected at purchase. Enterprise support includes priority assistance with migration, custom implementation guidance, and direct access to the engineering team.


# NL Layer
Source: https://docs.pandas-ai.com/v3/overview-nl

Understanding the AI and natural language processing capabilities of PandasAI

## How does PandasAI NL Layer work?

The Natural Language Layer uses generative AI to transform natural language queries into production-ready code generated by LLMs.
When you use the [`.chat`](/v3/chat-and-output) method on a dataframe, PandasAI passes to the LLM the question, the table headers, and 5-10 rows of the Dataframe.
It then instructs the LLM to generate the most relevant code, whether Python or SQL. The code is then executed locally.
There are different output formats supported by PandasAI, which can be found [here](/v3/chat-and-output#available-output-formats).

## Configure the NL Layer

PandasAI allows you to configure the NL Layer with the `config.set()` method.

Example:

```python
import pandasai as pai
from pandasai_litellm.litellm import LiteLLM

# Initialize LiteLLM with your OpenAI model
llm = LiteLLM(model="gpt-4.1-mini", api_key="YOUR_OPENAI_API_KEY")

pai.config.set({
   "llm": llm,
   "save_logs": True,
   "verbose": False,
   "max_retries": 3
})
```

### Parameters

#### llm

* **Description**: The LLM to use. You can pass an instance of an LLM or the name of an LLM. See [supported LLMs](/v3/large-language-models) for setup instructions and configuration options.

#### save\_logs

* **Type**: `bool`
* **Default**: `True`
* **Description**: Whether to save the logs of the LLM. You will find the logs in the `pandasai.log` file in the root of your project.

#### verbose

* **Type**: `bool`
* **Default**: `False`
* **Description**: Whether to print the logs in the console as PandasAI is executed.

#### max\_retries

* **Type**: `int`
* **Default**: `3`
* **Description**: The maximum number of retries to use when using the error correction framework. You can use this setting to override the default number of retries.


# Privacy & Security
Source: https://docs.pandas-ai.com/v3/privacy-security

Understanding security implications and sandbox options in PandasAI

## Code Execution and Sandbox Environment

PandasAI executes Python code that is generated by Large Language Models (LLMs). While this provides powerful data analysis capabilities, it's crucial to understand the security implications, especially in production use cases where your application might be exposed to potential malicious attacks.

### Why Use a Sandbox?

When building applications that allow users to interact with PandasAI, there's a potential risk that malicious users might attempt to manipulate the LLM into generating harmful code. To mitigate this risk, PandasAI provides a secure sandbox environment with the following features:

* **Isolated Execution**: Code runs in a completely isolated Docker container
* **Offline Operation**: The sandbox runs entirely offline, preventing any external network requests
* **Resource Limitations**: Strict controls on system resource usage
* **File System Isolation**: Protected access to the file system

### Using the Sandbox

To use the sandbox environment, you first need to install the required package and have Docker running on your system:

```bash
pip install pandasai-docker
```

<Note title="Sandbox Requirements">
  Make sure you have Docker running on your system before using the sandbox
  environment.
</Note>

Here's how to enable the sandbox for your PandasAI chat:

```python
import pandasai as pai
from pandasai_docker import DockerSandbox
from pandasai_litellm.litellm import LiteLLM

# Initialize LiteLLM with your OpenAI model
llm = LiteLLM(model="gpt-4.1-mini", api_key="YOUR_OPENAI_API_KEY")

# Configure PandasAI to use this LLM
pai.config.set({
    "llm": llm
})

# initialize the sandbox
sandbox = DockerSandbox()
sandbox.start()

# read a csv as df
df = pai.read_csv("./data/heart.csv")

# pass the df and the sandbox
result = pai.chat("plot total heart patients by gender", df, sandbox=sandbox)

# display the chart
result.show()

# stop the sandbox (docker container)
sandbox.stop()
```

### When to Use the Sandbox

We strongly recommend using the sandbox environment in the following scenarios:

* Building public-facing applications
* Processing untrusted user inputs
* Deploying in production environments
* Handling sensitive data
* Multi-tenant environments

### Enterprise Sandbox Options

For production-ready use cases, we offer several advanced sandbox options as part of our Enterprise license. These include:

* Custom security policies
* Advanced resource management
* Enhanced monitoring capabilities
* Additional isolation layers

See [Enterprise Features](/v3/enterprise-features) for more information about enterprise offerings. If you need assistance with implementation, please visit [pandas-ai.com](https://pandas-ai.com/). Our team can help you choose and configure the right security solution for your specific use case.


# DB Data Extensions
Source: https://docs.pandas-ai.com/v3/semantic-layer/data-ingestion

Learn how to ingest data from various sources in PandasAI

## What type of data does PandasAI support?

PandasAI mission is to make data analysis and manipulation more efficient and accessible to everyone. You can work with data in various ways:

* **CSV and Excel Files**: Load data directly from files using simple Python functions
* **SQL Databases**: Connect to various SQL databases using our extensions
* **Cloud Data**: Work with enterprise-scale data using our specialized extensions (requires [Enterprise License](/v3/enterprise-features))

Let's start with the basics of loading CSV files, and then we'll explore the different extensions available.

## How to work with CSV files in PandasAI?

Loading data from CSV files is straightforward with PandasAI:

```python
import pandasai as pai

# Basic CSV loading
file = pai.read_csv("data.csv")

# Use the semantic layer on CSV
df = pai.create(
    path="company/sales-data",
    df = file,
    description="Sales data from our retail stores",
    columns={
        "transaction_id": {"type": "string", "description": "Unique identifier for each sale"},
        "sale_date": {"type": "datetime", "description": "Date and time of the sale"},
        "product_id": {"type": "string", "description": "Product identifier"},
        "quantity": {"type": "integer", "description": "Number of units sold"},
        "price": {"type": "float", "description": "Price per unit"}
    },
)

# Chat with the dataframe
response = df.chat("Which product has the highest sales?")

```

## How to work with SQL in PandasAI?

PandasAI provides a sql extension for you to work with SQL, PostgreSQL, MySQL, CockroachDB, and Microsoft SQL Server databases.
To make the library lightweight and easy to use, the basic installation of the library does not include this extension.
It can be easily installed using pip with the specific database you want to use:

```bash
pip install pandasai-sql[postgres]
pip install pandasai-sql[mysql]
pip install pandasai-sql[cockroachdb]
pip install pandasai-sql[sqlserver]
```

Once you have installed the extension, you can use the [semantic data layer](/v3/semantic-layer#for-sql-databases-using-the-create-method) and perform [data transformations](/docs/v3/transformations).

```python
# MySQL example
sql_table = pai.create(
    path="example/mysql-dataset",
    description="Heart disease dataset from MySQL database",
    source={
        "type": "mysql",
        "connection": {
            "host": "database.example.com",
            "port": 3306,
            "user": "${DB_USER}",
            "password": "${DB_PASSWORD}",
            "database": "medical_data"
        },
        "table": "heart_data",
        "columns": [
            {"name": "Age", "type": "integer", "description": "Age of the patient in years"},
            {"name": "Sex", "type": "string", "description": "Gender of the patient (M = male, F = female)"},
            {"name": "ChestPainType", "type": "string", "description": "Type of chest pain (ATA, NAP, ASY, TA)"},
            {"name": "RestingBP", "type": "integer", "description": "Resting blood pressure in mm Hg"},
            {"name": "Cholesterol", "type": "integer", "description": "Serum cholesterol in mg/dl"},
            {"name": "FastingBS", "type": "integer", "description": "Fasting blood sugar > 120 mg/dl (1 = true, 0 = false)"},
            {"name": "RestingECG", "type": "string", "description": "Resting electrocardiogram results (Normal, ST, LVH)"},
            {"name": "MaxHR", "type": "integer", "description": "Maximum heart rate achieved"},
            {"name": "ExerciseAngina", "type": "string", "description": "Exercise-induced angina (Y = yes, N = no)"},
            {"name": "Oldpeak", "type": "float", "description": "ST depression induced by exercise relative to rest"},
            {"name": "ST_Slope", "type": "string", "description": "Slope of the peak exercise ST segment (Up, Flat, Down)"},
            {"name": "HeartDisease", "type": "integer", "description": "Heart disease diagnosis (1 = present, 0 = absent)"}
        ]
    }
)

# SQL Server example
sql_server_table = pai.create(
    path="example/sqlserver-dataset",
    description="Sales data from SQL Server database",
    source={
        "type": "sqlserver",
        "connection": {
            "host": "sqlserver.example.com",
            "port": 1433,
            "user": "${SQLSERVER_USER}",
            "password": "${SQLSERVER_PASSWORD}",
            "database": "sales_data"
        },
        "table": "transactions",
        "columns": [
            {"name": "transaction_id", "type": "string", "description": "Unique identifier for each transaction"},
            {"name": "customer_id", "type": "string", "description": "Customer identifier"},
            {"name": "transaction_date", "type": "datetime", "description": "Date and time of transaction"},
            {"name": "product_category", "type": "string", "description": "Product category"},
            {"name": "quantity", "type": "integer", "description": "Number of items sold"},
            {"name": "unit_price", "type": "float", "description": "Price per unit"},
            {"name": "total_amount", "type": "float", "description": "Total transaction amount"}
        ]
    }
)
```

## How to work with Enterprise Cloud Data in PandasAI?

PandasAI provides Enterprise Edition extensions for connecting to cloud data. These extensions require an [Enterprise License](/v3/enterprise-features).
Once you have installed a enterprise cloud data extension, you can use it to connect to your cloud data.

### Snowflake extension (ee)

First, install the extension:

```bash
poetry add pandasai-snowflake
# or
pip install pandasai-snowflake
```

Then use it:

```yaml
name: sales_data

source:
  type: snowflake
  connection:
    account: your-account
    warehouse: your-warehouse
    database: your-database
    schema: your-schema
    user: ${SNOWFLAKE_USER}
    password: ${SNOWFLAKE_PASSWORD}
  table: sales_data

destination:
  type: local
  format: parquet
  path: company/snowflake-sales

columns:
  - name: transaction_id
    type: string
    description: Unique identifier for each sale
  - name: sale_date
    type: datetime
    description: Date and time of the sale
  - name: product_id
    type: string
    description: Product identifier
  - name: quantity
    type: integer
    description: Number of units sold
  - name: price
    type: float
    description: Price per unit

transformations:
  - type: convert_timezone
    params:
      column: sale_date
      from: UTC
      to: America/Chicago
  - type: calculate
    params:
      column: revenue
      formula: quantity * price
  - type: round
    params:
      column: revenue
      decimals: 2

update_frequency: daily

order_by:
  - sale_date DESC

limit: 100000
```

### Databricks extension (ee)

First, install the extension:

```bash
poetry add pandasai-databricks
# or
pip install pandasai-databricks
```

Then use it:

```yaml
name: customer_data

source:
  type: databricks
  connection:
    host: your-workspace-url
    token: ${DATABRICKS_TOKEN}
  table: customers

destination:
  type: local
  format: parquet
  path: company/databricks-customers

columns:
  - name: customer_id
    type: string
    description: Unique identifier for each customer
  - name: name
    type: string
    description: Customer's full name
  - name: email
    type: string
    description: Customer's email address
  - name: join_date
    type: datetime
    description: Date when customer joined
  - name: total_purchases
    type: integer
    description: Total number of purchases made

transformations:
  - type: anonymize
    params:
      columns: [email, name]
  - type: convert_timezone
    params:
      column: join_date
      from: UTC
      to: Europe/London
  - type: calculate
    params:
      column: customer_tier
      formula: "CASE WHEN total_purchases > 100 THEN 'Gold' WHEN total_purchases > 50 THEN 'Silver' ELSE 'Bronze' END"

update_frequency: daily

order_by:
  - join_date DESC

limit: 100000
```

### BigQuery extension (ee)

First, install the extension:

```bash
poetry add pandasai-bigquery
# or
pip install pandasai-bigquery
```

Then use it:

```yaml
name: inventory_data

source:
  type: bigquery
  connection:
    project_id: your-project-id
    credentials: ${GOOGLE_APPLICATION_CREDENTIALS}
  table: inventory

destination:
  type: local
  format: parquet
  path: company/bigquery-inventory

columns:
  - name: product_id
    type: string
    description: Unique identifier for each product
  - name: product_name
    type: string
    description: Name of the product
  - name: category
    type: string
    description: Product category
  - name: stock_level
    type: integer
    description: Current quantity in stock
  - name: last_updated
    type: datetime
    description: Last inventory update timestamp

transformations:
  - type: categorize
    params:
      column: stock_level
      bins: [0, 20, 100, 500]
      labels: ["Low", "Medium", "High"]
  - type: extract
    params:
      column: product_name
      pattern: "(.*?)\\s*-\\s*(.*)"
      into: [brand, model]
  - type: convert_timezone
    params:
      column: last_updated
      from: UTC
      to: Asia/Tokyo

update_frequency: hourly

order_by:
  - last_updated DESC

limit: 50000
```

### Oracle extension (ee)

First, install the extension:

```bash
poetry add pandasai-oracle
# or
pip install pandasai-oracle
```

Then use it:

```yaml
name: sales_data

source:
  type: oracle
  connection:
    host: your-host
    port: 1521
    service_name: your-service
    user: ${ORACLE_USER}
    password: ${ORACLE_PASSWORD}
  table: sales_data

destination:
  type: local
  format: parquet
  path: company/oracle-sales

columns:
  - name: transaction_id
    type: string
    description: Unique identifier for each sale
  - name: sale_date
    type: datetime
    description: Date and time of the sale
  - name: product_id
    type: string
    description: Product identifier
  - name: quantity
    type: integer
    description: Number of units sold
  - name: price
    type: float
    description: Price per unit

transformations:
  - type: convert_timezone
    params:
      column: sale_date
      from: UTC
      to: Australia/Sydney
  - type: calculate
    params:
      column: total_amount
      formula: quantity * price
  - type: round
    params:
      column: total_amount
      decimals: 2
  - type: calculate
    params:
      column: discount
      formula: "CASE WHEN quantity > 10 THEN 0.1 WHEN quantity > 5 THEN 0.05 ELSE 0 END"

update_frequency: daily

order_by:
  - sale_date DESC

limit: 100000
```

### Yahoo Finance extension

First, install the extension:

```bash
poetry add pandasai-yfinance
# or
pip install pandasai-yfinance
```

Then use it:

```yaml
name: stock_data

source:
  type: yahoo_finance
  symbols: 
    - GOOG
    - MSFT
    - AAPL
  start_date: 2023-01-01
  end_date: 2023-12-31

destination:
  type: local
  format: parquet
  path: company/market-data

columns:
  - name: date
    type: datetime
    description: Date of the trading day
  - name: open
    type: float
    description: Opening price of the stock
  - name: high
    type: float
    description: Highest price of the stock during the day
  - name: low
    type: float
    description: Lowest price of the stock during the day
  - name: close
    type: float
    description: Closing price of the stock
  - name: volume
    type: integer
    description: Number of shares traded during the day

transformations:
  - type: calculate
    params:
      column: daily_return
      formula: (close - open) / open * 100
  - type: calculate
    params:
      column: price_range
      formula: high - low
  - type: round
    params:
      columns: [daily_return, price_range]
      decimals: 2
  - type: convert_timezone
    params:
      column: date
      from: UTC
      to: America/New_York

update_frequency: daily

order_by:
  - date DESC

limit: 100000
```

## All data extensions

<table style={{ borderCollapse: 'collapse', width: '100%', border: '1px solid #ccc' }}>
  <tr>
    <th style={{ border: '1px solid #ccc', padding: '8px 16px', textAlign: 'left' }}>extension</th>
    <th style={{ border: '1px solid #ccc', padding: '8px 16px', textAlign: 'left' }}>install with poetry</th>
    <th style={{ border: '1px solid #ccc', padding: '8px 16px', textAlign: 'left' }}>install with pip</th>
    <th style={{ border: '1px solid #ccc', padding: '8px 16px', textAlign: 'left' }}>need ee license?</th>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai\_sql</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>poetry add pandasai-sql\[postgres|mysql|cockroachdb|sqlserver]</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>pip install pandasai-sql\[postgres|mysql|cockroachdb|sqlserver]</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>No</td>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai\_yfinance</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>poetry add pandasai-yfinance</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>pip install pandasai-yfinance</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>No</td>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai\_snowflake</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>poetry add pandasai-snowflake</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>pip install pandasai-snowflake</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Yes</td>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai\_databricks</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>poetry add pandasai-databricks</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>pip install pandasai-databricks</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Yes</td>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai\_bigquery</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>poetry add pandasai-bigquery</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>pip install pandasai-bigquery</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Yes</td>
  </tr>

  <tr>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>pandasai\_oracle</td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>poetry add pandasai-oracle</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}><code>pip install pandasai-oracle</code></td>
    <td style={{ border: '1px solid #ccc', padding: '8px 16px' }}>Yes</td>
  </tr>
</table>


# Create a New Schema
Source: https://docs.pandas-ai.com/v3/semantic-layer/new

Create a new semantic layer schema using the `create` method

<Note title="Beta Notice">
  The semantic data layer is an experimental feature, suggested to advanced users.
</Note>

### Using the `pai.create()` method with CSV and parquet files

The simplest way to define a semantic layer schema is using the `create` method:

```python
import pandasai as pai

# Load your data: for example, in this case, a CSV
file = pai.read_csv("data.csv")

df = pai.create(
    # Format: "organization/dataset"
    path="company/sales-data",

    # Input dataframe
    df = file,

    # Optional description
    description="Sales data from our retail stores",

    # Define the structure and metadata of your dataset's columns.
    # If not provided, all columns from the input dataframe will be included.
    columns=[
        {
            "name": "transaction_id",
            "type": "string",
            "description": "Unique identifier for each sale"
        },
        {
            "name": "sale_date"
            "type": "datetime",
            "description": "Date and time of the sale"
        }
    ]
)
```

#### - path

The path uniquely identifies your dataset in the PandasAI ecosystem using the format "organization/dataset".

```python
file = pai.read_csv("data.csv")

pai.create(
    path="acme-corp/sales-data",  # Format: "organization/dataset"
    ...
)
```

**Type**: `str`

* Must follow the format: "organization-identifier/dataset-identifier"
* Organization identifier should be unique to your organization
* Dataset identifier should be unique within your organization
* Examples: "acme-corp/sales-data", "my-org/customer-profiles"

#### - df

The input dataframe that contains your data, typically created using `pai.read_csv()`.

```python
file = pai.read_csv("data.csv")  # Create the input dataframe

pai.create(
    path="acme-corp/sales-data",
    df=file,  # Pass your dataframe here
    ...
)
```

**Type**: `DataFrame`

* Must be a pandas DataFrame created with `pai.read_csv()`
* Contains the raw data you want to enhance with semantic information
* Required parameter for creating a semantic layer

#### - description

A clear text description that helps others understand the dataset's contents and purpose.

```python
file = pai.read_csv("data.csv")

pai.create(
    path="company/sales-data",
    df = file,
    description="Daily sales transactions from all retail stores, including transaction IDs, dates, and amounts",
    ...
)
```

**Type**: `str`

* The purpose of the dataset
* The type of data contained
* Any relevant context about data collection or usage
* Optional but recommended for better data understanding

#### - columns

Define the structure and metadata of your dataset's columns to help PandasAI understand your data better.

**Note**: If the `columns` parameter is not provided, all columns from the input dataframe will be included in the semantic layer.
When specified, only the declared columns will be included, allowing you to select specific columns for your semantic layer.

```python
file = pai.read_csv("data.csv")

pai.create(
    path="company/sales-data",
    df = file,
    description="Daily sales transactions from all retail stores",
    columns=[
        {
            "name": "transaction_id",
            "type": "string",
            "description": "Unique identifier for each sale"
        },
        {
            "name": "sale_date"
            "type": "datetime",
            "description": "Date and time of the sale"
        },
        {
            "name": "quantity",
            "type": "integer",
            "description": "Number of units sold"
        },
        {
            "name": "price",
            "type": "float",
            "description": "Price per unit in USD"
        },
        {
            "name": "is_online",
            "type": "boolean",
            "description": "Whether the sale was made online"
        }
    ]
)
```

**Type**: `dict[str, dict]`

* Keys: column names as they appear in your DataFrame
* Values: dictionary containing:
  * `type` (str): Data type of the column
    * "string": IDs, names, categories
    * "integer": counts, whole numbers
    * "float": prices, percentages
    * "datetime": timestamps, dates
    * "boolean": flags, true/false values
  * `description` (str): Clear explanation of what the column represents

### Using the `pai.create()` method for SQL databases

<Note title="Extra Dependency Required">
  You need to install the `pandasai-sql` extra dependency for this feature.
  See [SQL installation instructions](/v3/data-ingestion#how-to-work-with-sql-in-PandasAI).
</Note>

For SQL databases, you can use the `create` method to define your data source and schema. Here's an example using a MySQL database:

```python
sql_table = pai.create(
    # Format: "organization/dataset"
    path="company/health-data",

    # Optional description
    description="Heart disease dataset from MySQL database",

    # Define the source of the data, including connection details and
    # table name
    source={
        "type": "mysql",
        "connection": {
            "host": "${DB_HOST}",
            "port": 3306,
            "user": "${DB_USER}",
            "password": "${DB_PASSWORD}",
            "database": "${DB_NAME}"
        },
        "table": "heart_data"
    }
)
```

In this example:

* The `path` defines where the dataset will be stored in your project
* The `description` provides context about the dataset
* The `source` object contains:
  * Database connection details (using environment variables for security)
  * Table name to query
  * Column definitions with types and descriptions

<Note>
  For security best practices, always use environment variables for sensitive connection details. Never hardcode credentials in your code.
</Note>

You can then use this dataset like any other:

```python
# Load the dataset
heart_data = pai.load("organization/health-data")

# Query the data
response = heart_data.chat("What is the average age of patients with heart disease?")
```

### YAML Semantic Layer Configuration

Whenever you create a semantic layer schema using the `create` method, a YAML configuration file is automatically generated for you in the `datasets/` directory of your project.
As an alternative, you can use a YAML `schema.yaml` file directly in the `datasets/organization_name/dataset_name` directory.

The following sections detail all available configuration options for your schema.yaml file:

#### - description

A clear text description that helps others understand the dataset's contents and purpose.

**Type**: `str`

* The purpose of the dataset, in order for everyone in the organization and for the LLMs to understand

```yaml
description: Daily sales transactions from all retail stores, including transaction IDs, dates, and amounts
```

#### - source (mandatory for SQL datasets)

Specify the data source for your dataset.

```yaml
source:
  type: postgres
  connection:
    host: postgres-host
    port: 5432
    database: postgres
    user: postgres
    password: ******
  table: orders
  view: false
```

> The available data sources depends on the installed data extensions (sql databases, data lakehouses, yahoo\_finance).

**Type**: `dict`

* `type` (str): Type of data source
  * "postgresql" for PostgreSQL databases
  * "mysql" for MySQL databases
  * "bigquery" for Google BigQuery data
  * "snowflake" for Snowflake data
  * "databricks" for Databricks data
  * "oracle" for Oracle databases
  * "yahoo\_finance" for Yahoo Finance data
* `connection_string` (str): Connection string for the data source
* `query` (str): Query to retrieve data from the data source

#### - columns

Define the structure and metadata of your dataset's columns to help PandasAI understand your data better.

```yaml
columns:
  - name: transaction_id
    type: string
    description: Unique identifier for each sale
  - name: sale_date
    type: datetime
    description: Date and time of the sale
```

**Type**: `list[dict]`

* Each dictionary represents a column.
* **Fields**:
  * `name` (str): Name of the column.
    * For tables: Use simple column names (e.g., `transaction_id`).
  * `type` (str): Data type of the column.
    * Supported types:
      * `"string"`: IDs, names, categories.
      * `"integer"`: Counts, whole numbers.
      * `"float"`: Prices, percentages.
      * `"datetime"`: Timestamps, dates.
      * `"boolean"`: Flags, true/false values.
  * `description` (str): Clear explanation of what the column represents.

**Constraints**:

1. Column names must be unique.
2. For views, all column names must be in the format `[table].[column]`.

#### - transformations

Apply transformations to your data to clean, convert, or anonymize it.

```yaml
transformations:
  - type: anonymize
    params:
      columns:
        - transaction_id
      method: hash
  - type: convert_timezone
    params:
      columns:
        - sale_date
      from_timezone: UTC
      to_timezone: America/New_York
```

**Type**: `list[dict]`

* Each dictionary represents a transformation
* `type` (str): Type of transformation
  * "anonymize" for anonymizing data
  * "convert\_timezone" for converting timezones
* `params` (dict): Parameters for the transformation

> If you want to learn more about transformations, check out the [transformations documentation](/v3/transformations).

### Group By Configuration

The `group_by` field allows you to specify which columns can be used for grouping operations. This is particularly useful for aggregation queries and data analysis.

```yaml
columns:
  - name: order.date
    type: datetime
    description: Date and time of the sale
  ...
group_by:
  - order.date
  - order.status
```

**Configuration Options:**

* `group_by` (list\[str]):
  * List of column references in the format `table.column`
  * Specifies which columns can be used for grouping operations
  * Can reference any column from any table in your schema

### Column expressions and aliases

The `expression` field allows you to specify a SQL expression for a column. This expression will be used in the query instead of the column name.

```yaml
columns:
  - name: transaction_amount
    type: float
    description: Amount of the transaction
    alias: amount
  - name: total_revenue
    type: float
    description: Total revenue including tax
    expression: "transaction_amount * (1 + tax_rate)"
    alias: revenue
```

**Configuration Options:**

* `alias` (str):
  * Alternative name that can be used to reference the column
  * Useful for supporting different naming conventions or more intuitive names
  * Must be unique across all columns and their aliases

* `expression` (str):
  * Formula for calculating derived columns
  * Uses other column names as variables
  * Supports basic arithmetic operations (+, -, \*, /)
  * Can reference other columns in the same schema

**Best Practices:**

* Keep aliases concise and descriptive
* Avoid using special characters or spaces in aliases
* Use consistent naming conventions
* Document the purpose of derived columns in their description


# Semantic Data Layer
Source: https://docs.pandas-ai.com/v3/semantic-layer/semantic-layer

Turn raw data into semantic-enhanced and clean dataframes

<Note title="Experimental Feature">
  The semantic data layer is an experimental feature, suggested to advanced users.
</Note>

PandasAI 3.0 introduces a new feature: the semantic layer, which allows you to turn raw data into semantic-enhanced and clean dataframes, making it easier to work with and analyze your data.

## What's the Semantic Layer?

The semantic layer allows you to turn raw data into dataframes you can ask questions to as conversational AI dashboards. It serves several important purposes:

1. **Data configuration**: Define how your data should be loaded and processed
2. **Semantic information**: Add context and meaning to your data columns
3. **Data transformation**: Specify how data should be cleaned and transformed

## How to start using the Semantic Layer?

In order to use the semantic layer, you need to create a new schema for each dataset you want to work with.
If you want to learn more about how to create a semantic layer schema, check out [how to create a semantic layer schema](/v3/semantic-layer/new).


# Data Transformations
Source: https://docs.pandas-ai.com/v3/semantic-layer/transformations

Available data transformations in PandasAI

<Note title="Beta Notice">
  The semantic data layer is an experimental feature, suggested to advanced users.
</Note>

## Data Transformations in PandasAI

PandasAI provides a rich set of data transformations that can be applied to your data. These transformations can be specified in your schema file or applied programmatically.

### String Transformations

```yaml
transformations:
  # Convert text to lowercase
  - type: to_lowercase
    params:
      column: product_name

  # Convert text to uppercase
  - type: to_uppercase
    params:
      column: category

  # Remove leading/trailing whitespace
  - type: strip
    params:
      column: description

  # Truncate text to specific length
  - type: truncate
    params:
      column: description
      length: 100
      add_ellipsis: true  # Optional, adds "..." to truncated text

  # Pad strings to fixed width
  - type: pad
    params:
      column: product_code
      width: 10
      side: left  # Optional: "left" or "right", default "left"
      pad_char: "0"  # Optional, default " "

  # Extract text using regex
  - type: extract
    params:
      column: product_code
      pattern: "^[A-Z]+-(\d+)"  # Extracts numbers after hyphen
```

### Numeric Transformations

```yaml
transformations:
  # Round numbers to specified decimals
  - type: round_numbers
    params:
      column: price
      decimals: 2

  # Scale values by a factor
  - type: scale
    params:
      column: price
      factor: 1.1  # 10% increase

  # Clip values to bounds
  - type: clip
    params:
      column: quantity
      lower: 0  # Optional
      upper: 100  # Optional

  # Normalize to 0-1 range
  - type: normalize
    params:
      column: score

  # Standardize using z-score
  - type: standardize
    params:
      column: score

  # Ensure positive values
  - type: ensure_positive
    params:
      column: amount
      drop_negative: false  # Optional, drops rows with negative values if true

  # Bin continuous data
  - type: bin
    params:
      column: age
      bins: [0, 18, 35, 50, 65, 100]  # Or specify number of bins: bins: 5
      labels: ["0-18", "19-35", "36-50", "51-65", "65+"]  # Optional
```

### Date and Time Transformations

```yaml
transformations:
  # Convert timezone
  - type: convert_timezone
    params:
      column: timestamp
      to: "US/Pacific"

  # Format dates
  - type: format_date
    params:
      column: date
      format: "%Y-%m-%d"

  # Convert to datetime
  - type: to_datetime
    params:
      column: date
      format: "%Y-%m-%d"  # Optional
      errors: "coerce"  # Optional: "raise", "coerce", or "ignore"

  # Validate date range
  - type: validate_date_range
    params:
      column: date
      start_date: "2024-01-01"
      end_date: "2024-12-31"
      drop_invalid: false  # Optional
```

### Data Cleaning Transformations

```yaml
transformations:
  # Fill missing values
  - type: fill_na
    params:
      column: quantity
      value: 0

  # Replace values
  - type: replace
    params:
      column: status
      old_value: "inactive"
      new_value: "disabled"

  # Remove duplicates
  - type: remove_duplicates
    params:
      columns: ["order_id", "product_id"]
      keep: "first"  # Optional: "first", "last", or false

  # Normalize phone numbers
  - type: normalize_phone
    params:
      column: phone
      country_code: "+1"  # Optional, default "+1"
```

### Categorical Transformations

```yaml
transformations:
  # One-hot encode categories
  - type: encode_categorical
    params:
      column: category
      drop_first: true  # Optional

  # Map values using dictionary
  - type: map_values
    params:
      column: grade
      mapping:
        "A": 4.0
        "B": 3.0
        "C": 2.0

  # Standardize categories
  - type: standardize_categories
    params:
      column: company
      mapping:
        "Apple Inc.": "Apple"
        "Apple Computer": "Apple"
```

### Rename Column

Renames a column to a new name.

**Parameters:**

* `column` (str): The current column name
* `new_name` (str): The new name for the column

**Example:**

```yaml
transformations:
  - type: rename
    params:
      column: old_name
      new_name: new_name
```

This will rename the column `old_name` to `new_name`.

### Validation Transformations

```yaml
transformations:
  # Validate email format
  - type: validate_email
    params:
      column: email
      drop_invalid: false  # Optional

  # Validate foreign key references
  - type: validate_foreign_key
    params:
      column: user_id
      ref_df: users  # Reference DataFrame
      ref_column: id
      drop_invalid: false  # Optional
```

### Privacy and Security Transformations

```yaml
transformations:
  # Anonymize sensitive data
  - type: anonymize
    params:
      column: email  # Replaces username in emails with asterisks
```

## Type Conversion Transformations

```yaml
transformations:
  # Convert to numeric type
  - type: to_numeric
    params:
      column: amount
      errors: "coerce"  # Optional: "raise", "coerce", or "ignore"
```

## Chaining Transformations

You can chain multiple transformations in sequence. The transformations will be applied in the order they are specified:

```yaml
transformations:
  - type: to_lowercase
    params:
      column: product_name
  - type: strip
    params:
      column: product_name
  - type: truncate
    params:
      column: product_name
      length: 50
```

## Programmatic Usage

While schema files are convenient for static transformations, you can also apply transformations programmatically using the `TransformationManager`:

```python
import pandasai as pai

df = pai.read_csv("data.csv")
manager = TransformationManager(df)
result = (manager
    .validate_email("email", drop_invalid=True)
    .normalize_phone("phone")
    .validate_date_range("birth_date", "1900-01-01", "2024-01-01")
    .remove_duplicates("user_id")
    .ensure_positive("amount")
    .standardize_categories("company", {"Apple Inc.": "Apple"})
    .df)
```

This approach allows for a fluent interface, chaining multiple transformations together. Each method returns the manager instance, enabling further transformations. The final `.df` attribute returns the transformed DataFrame.

## Complete Example

Let's walk through a complete example of data transformation using a sales dataset. This example demonstrates how to clean, validate, and prepare your data for analysis.

### Sample Data

Consider a CSV file `sales_data.csv` with the following structure:

```csv
date,store_id,product_name,category,quantity,unit_price,customer_email
2024-01-15, ST001,  iPhone 13 Pro,Electronics,2,999.99,john.doe@email.com
2024-01-15,ST002,macBook Pro ,Electronics,-1,1299.99,invalid.email
2024-01-16,ST001,AirPods Pro,Electronics,3,249.99,jane@example.com
2024-01-16,ST003,iMac 27" ,Electronics,1,1799.99,
```

### Schema File

Create a `schema.yaml` file to define the transformations:

```yaml
name: sales_data
description: "Daily sales data from retail stores"
source:
  type: csv
  path: "sales_data.csv"

transformations:
  # Clean up product names
  - type: strip
    params:
      column: product_name
  - type: standardize_categories
    params:
      column: product_name
      mapping:
        "iPhone 13 Pro": "iPhone 13 Pro"
        "macBook Pro": "MacBook Pro"
        "AirPods Pro": "AirPods Pro"
        "iMac 27\"": "iMac 27-inch"

  # Format dates
  - type: to_datetime
    params:
      column: date
      format: "%Y-%m-%d"

  # Validate and clean store IDs
  - type: pad
    params:
      column: store_id
      width: 5
      side: "right"
      pad_char: "0"

  # Ensure valid quantities
  - type: ensure_positive
    params:
      column: quantity
      drop_negative: true

  # Format prices
  - type: round_numbers
    params:
      column: unit_price
      decimals: 2

  # Validate emails
  - type: validate_email
    params:
      column: customer_email
      drop_invalid: false

  # Add derived columns
  - type: scale
    params:
      column: unit_price
      factor: 1.1  # Add 10% tax

columns:
  date:
    type: datetime
    description: "Date of sale"
  store_id:
    type: string
    description: "Store identifier"
  product_name:
    type: string
    description: "Product name"
  category:
    type: string
    description: "Product category"
  quantity:
    type: integer
    description: "Number of units sold"
  unit_price:
    type: float
    description: "Price per unit"
  customer_email:
    type: string
    description: "Customer email address"
```

### Python Code

Here's how to use the schema and transformations in your code:

```python
import pandasai as pai

# Load and transform the data of the schema we just created
df = pai.load("my-org/sales-data")

# The resulting DataFrame will have:
# - Cleaned and standardized product names
# - Properly formatted dates
# - Padded store IDs (e.g., "ST001000")
# - Only positive quantities
# - Rounded prices with tax
# - Validated email addresses

# You can now analyze the data
response = df.chat("What's our best-selling product?")

# Or export the transformed data
df.to_csv("cleaned_sales_data.csv")
```

### Result

The transformed data will look like this:

```csv
date,store_id,product_name,category,quantity,unit_price,customer_email,email_valid
2024-01-15,ST001000,iPhone 13 Pro,Electronics,2,1099.99,john.doe@email.com,true
2024-01-16,ST001000,AirPods Pro,Electronics,3,274.99,jane@example.com,true
2024-01-16,ST003000,iMac 27-inch,Electronics,1,1979.99,,false
```

Notice how the transformations have:

* Standardized product names
* Padded store IDs
* Removed negative quantity rows
* Added 10% tax to prices
* Validated email addresses
* Added an email validation column

This example demonstrates how to use multiple transformations together to clean and prepare your data for analysis. The transformations are applied in sequence, and each transformation builds on the results of the previous ones.


# Data Views
Source: https://docs.pandas-ai.com/v3/semantic-layer/views

Learn how to work with views in PandasAI

<Note title="Beta Notice">
  The semantic data layer is an experimental feature, suggested to advanced users.
</Note>

## What are Views?

Views are a feature of SQL databases that allow you to define logical subsets of data that can be used in queries. In PandasAI, you can define views in your semantic layer schema to organize and structure your data. Views are particularly useful when you want to:

* Combine data from multiple datasets
* Create a simplified or filtered view of your data
* Define relationships between different datasets

## Creating Views

You can create views either through YAML configuration or programmatically using Python.

### Python Code Example

```python
import pandasai as pai

# Create source datasets for an e-commerce analytics system
# Orders dataset
orders_df = pai.read_csv("orders.csv")
orders_dataset = pai.create(
    "myorg/orders",
    orders_df,
    description="Customer orders and transaction data"
)

# Products dataset
products_df = pai.read_csv("products.csv")
products_dataset = pai.create(
    "myorg/products",
    products_df,
    description="Product catalog with categories and pricing"
)

# Customer dataset
customers_df = pai.read_csv("customers.csv")
customers_dataset = pai.create(
    "myorg/customers",
    customers_df,
    description="Customer demographics and preferences"
)

# Define relationships between datasets
view_relations = [
    {
        "name": "order_to_product",
        "description": "Links orders to their products",
        "from": "orders.product_id",
        "to": "products.id"
    },
    {
        "name": "order_to_customer",
        "description": "Links orders to customer profiles",
        "from": "orders.customer_id",
        "to": "customers.id"
    }
]

# Select relevant columns for the sales analytics view
view_columns = [
    # Order details
    {"name": "orders.id", "type": "integer"},
    {"name": "orders.order_date", "type": "date"},
    {"name": "orders.total_amount", "type": "float"},
    {"name": "orders.status", "type": "string"},

    # Product information
    {"name": "products.name", "type": "string"},
    {"name": "products.category", "type": "string"},
    {"name": "products.unit_price", "type": "float"},
    {"name": "products.stock_level", "type": "integer"},

    # Customer information
    {"name": "customers.segment", "type": "string"},
    {"name": "customers.country", "type": "string"},
    {"name": "customers.join_date", "type": "date"},
]

# Create a comprehensive sales analytics view
sales_view = pai.create(
    "myorg/sales-analytics",
    description="Unified view of sales data combining orders, products, and customer information",
    relations=view_relations,
    columns=view_columns,
    view=True
)

# This view enables powerful analytics queries like:
# - Sales trends by customer segment and product category
# - Customer purchase history and preferences
# - Inventory management based on order patterns
# - Geographic sales distribution
```

### YAML Configuration

### Example Configuration

```yaml
name: table_heart
columns:
  - name: parents.id
  - name: parents.name
  - name: parents.age
  - name: children.name
  - name: children.age
relations:
  - name: parent_to_children
    description: Relation linking the parent to its children
    from: parents.id
    to: children.id
```

***

#### Constraints

1. **Mutual Exclusivity**:

   * A schema cannot define both `table` and `view` simultaneously.
   * If `view` is `true`, then the schema represents a view.

2. **Column Format**:

   * For views:
     * All columns must follow the format `[table].[column]`.
     * `from` and `to` fields in `relations` must follow the `[table].[column]` format.
     * Example: `loans.payment_amount`, `heart.condition`.

3. **Relationships for Views**:

   * Each table referenced in `columns` must have at least one relationship defined in `relations`.
   * Relationships must specify `from` and `to` attributes in the `[table].[column]` format.
   * Relations define how different tables in your view are connected.

4. **Dataset Requirements**:
   * All referenced datasets must exist before creating the view.
   * The columns specified in the view must exist in their respective source datasets.
   * The columns used in relations (`from` and `to`) must be compatible types.


# Skills
Source: https://docs.pandas-ai.com/v3/skills

Learn how to create and use custom skills to extend PandasAI's capabilities

<Note title="Enterprise Feature">
  Skills require a PandasAI Enterprise license. See [Enterprise Features](/v3/enterprise-features) for more details or [contact us](https://pandas-ai.com/) for production use.
</Note>

Skills allow you to add custom functions on a **global level** that extend PandasAI's capabilities beyond standard data analysis. Once a skill is defined using the `@pai.skill()` decorator, it becomes automatically available across your entire application - whether you're using `pai.chat()`, `SmartDataframe`, or `Agent`. These custom functions are registered globally and can be used by any PandasAI interface without additional configuration.

## Creating a Skill

Skills are created by decorating a Python function with `@pai.skill()`. The function should include clear documentation with type hints and a descriptive docstring, as the AI uses this information to understand when and how to use the skill.

### Basic Skill Definition

```python
import pandasai as pai

@pai.skill()
def my_custom_function(param1: str, param2: int) -> str:
    """
    A custom function that demonstrates skill creation.

    Args:
        param1 (str): First parameter description
        param2 (int): Second parameter description

    Returns:
        str: Result description
    """
    return f"Processed {param1} with value {param2}"
```

### Example Skills

Here are some practical examples of skills you can create:

```python
import pandasai as pai

@pai.skill()
def calculate_bonus(salary: float, performance: float) -> float:
    """
    Calculates employee bonus based on salary and performance score.

    Args:
        salary (float): Employee's base salary
        performance (float): Performance score (0-100)

    Returns:
        float: Calculated bonus amount
    """
    if performance >= 90:
        return salary * 0.15  # 15% bonus for excellent performance
    elif performance >= 70:
        return salary * 0.10  # 10% bonus for good performance
    else:
        return salary * 0.05  # 5% bonus for average performance

@pai.skill()
def plot_salaries(names: list[str], salaries: list[float]):
    """
    Creates a bar chart showing employee salaries.

    Args:
        names (list[str]): List of employee names
        salaries (list[float]): List of corresponding salaries
    """
    import matplotlib.pyplot as plt

    plt.figure(figsize=(10, 6))
    plt.bar(names, salaries)
    plt.xlabel("Employee Name")
    plt.ylabel("Salary ($)")
    plt.title("Employee Salaries")
    plt.xticks(rotation=45)
    plt.tight_layout()
    plt.show()

@pai.skill()
def format_currency(amount: float) -> str:
    """
    Formats a number as currency.

    Args:
        amount (float): The amount to format

    Returns:
        str: Formatted currency string
    """
    return f"${amount:,.2f}"
```

## Skills in Action

Once skills are defined, they are automatically available to all PandasAI interfaces. Here's how to use them with different components:

### Skills with pai.chat

```python
import pandasai as pai

# Skills are automatically registered when defined
@pai.skill()
def get_employee_stats(employee_id: int) -> dict:
    """
    Gets comprehensive statistics for an employee.

    Args:
        employee_id (int): The employee ID

    Returns:
        dict: Employee statistics including salary, bonus, and performance
    """
    # Your logic to fetch employee data
    return {
        "id": employee_id,
        "salary": 60000,
        "bonus": 9000,
        "performance": 92
    }

# Use pai.chat with the skill automatically available
response = pai.chat("Get statistics for employee ID 1 and calculate their total compensation")
# The AI will use both get_employee_stats() and calculate_bonus() skills
print(response)
```

### Skills with Agent

```python
import pandas as pd
import pandasai as pai
from pandasai import Agent
from pandasai_litellm.litellm import LiteLLM

# Add your model
llm = LiteLLM(model="ollama/llama3", api_base="http://localhost:11434/api/generate")

pai.config.set({"llm": llm})

# Sample employee data
employees_data = {
    "EmployeeID": [1, 2, 3, 4, 5],
    "Name": ["John", "Emma", "Liam", "Olivia", "William"],
    "Department": ["HR", "Sales", "IT", "Marketing", "Finance"],
    "Salary": [50000, 60000, 70000, 55000, 65000],
    "Performance": [85, 92, 78, 88, 95]
}

salaries_data = {
    "EmployeeID": [1, 2, 3, 4, 5],
    "Bonus": [7500, 9000, 7000, 5500, 9750]
}

employees_df = pai.DataFrame(employees_data)
salaries_df = pai.DataFrame(salaries_data)

# Create an agent with the dataframes
agent = Agent([employees_df, salaries_df], memory_size=10)

# Chat with the agent - skills are automatically available
response1 = agent.chat("Calculate bonuses for all employees and show the results")
print("Response 1:", response1)

response2 = agent.chat("Show me the total bonus amount formatted as currency")
print("Response 2:", response2)

# The agent can use multiple skills in one conversation
response3 = agent.chat("Calculate bonuses, format them as currency, and create a chart")
print("Response 3:", response3)
```