CSV Data Import

Import structured tabular data from CSV files to provide your AI agents with searchable databases, product catalogs, pricing information, and other structured datasets.

Overview

Property	Details
Type	Static
Refresh	Manual
Tier	2 (Professional & Enterprise)
Format	CSV with any column structure
Encoding	UTF-8 recommended

When to Use Data CSV

The Data CSV connector is ideal for:

• Product Catalogs - Product names, descriptions, SKUs, pricing

• Employee Directories - Staff information and contact details

• Inventory Data - Stock levels, warehouse locations

• Pricing Tables - Service tiers, feature matrices

• Location Data - Store addresses, hours, contact information

• Statistical Data - Reports, metrics, KPIs

Difference from QnA CSV

Feature	QnA CSV	Data CSV
Purpose	Question-Answer pairs	Structured tabular data
Columns	Must have Question & Answer	Any column structure
Use Case	FAQs, knowledge base	Databases, catalogs, lists
Tier	1 (All plans)	2 (Professional+)

CSV File Format

Flexible Column Structure

Unlike QnA CSV, Data CSV accepts any column structure:

ProductName,SKU,Price,Category,InStock
Widget Pro,WGT-001,99.99,Electronics,Yes
Gadget Plus,GDG-002,149.99,Electronics,No
Service Bundle,SVC-100,299.99,Services,Yes

Multi-Column Examples

Employee Directory:

Name,Department,Email,Phone,Location
John Smith,Engineering,[email protected],555-0100,Building A
Jane Doe,Sales,[email protected],555-0101,Building B

Store Locations:

StoreName,Address,City,State,Zip,Phone,Hours
Main Street Store,123 Main St,Springfield,IL,62701,555-1000,Mon-Fri 9AM-6PM
Downtown Branch,456 Oak Ave,Capital City,IL,62702,555-2000,Mon-Sat 8AM-8PM

Pricing Table:

Plan,MonthlyPrice,AnnualPrice,Users,Storage,Support
Starter,29,290,5,10GB,Email
Professional,99,990,25,100GB,Email+Chat
Enterprise,299,2990,Unlimited,Unlimited,24/7 Phone

How to Add Data CSV

Step 1: Navigate to Data Sources

1. Log in to your Twig AI account

2. Click Data in the main navigation menu

3. Click Add Data Source or the + button

Step 2: Select Data CSV Connector

1. Choose Data as a CSV from the list

2. The connector shows: "Tabular data as a CSV file"

3. Note: Available on Professional and Enterprise plans

Step 3: Configure the Data Source

Basic Information

• Name (required): Descriptive name for your dataset

  • Example: "Product Catalog 2024", "Employee Directory", "Store Locations"

• Description (optional): Context about the data

  • Example: "Complete product catalog with pricing and availability"

File Upload

1. Click Choose File or drag-and-drop your CSV file

2. Upload your .csv file

3. Wait for upload to complete

Tags (Optional)

• Add tags for organization

• Examples: "product", "pricing", "internal", "public"

Step 4: Save and Process

1. Click Save or Create

2. System will parse and index your tabular data

3. Monitor the processing status

Step 5: Verify Import

1. Check record count matches CSV rows

2. Verify status shows "END_PROCESS"

3. Test by asking questions about the data

Creating Effective Data CSVs

1. Use Clear Column Headers

Good:

ProductName,Description,Price,Availability

Avoid:

col1,col2,col3,col4
p_name,desc,$$,avail

2. Consistent Data Formatting

Good:

Product,Price,Date
Widget A,99.99,2024-01-15
Widget B,149.99,2024-01-16

Avoid:

Product,Price,Date
Widget A,$99.99,Jan 15 2024
Widget B,149.99 USD,2024/01/16

3. Include Descriptive Information

Good:

ProductName,Description,Features,Benefits
Premium Widget,"High-quality widget with advanced features","Durable, Fast, Efficient","Saves time, Reduces costs"

Better:

ProductName,FullDescription
Premium Widget,"The Premium Widget is a high-quality solution designed for professional use. Features include durable construction, fast processing, and efficient operation. Benefits: Saves time through automation, Reduces operational costs by 30%."

4. Handle Missing Data

Use consistent representations for missing data:

Product,Price,DiscountPrice,Notes
Widget A,99.99,79.99,On sale this week
Widget B,149.99,,Regular price
Widget C,199.99,N/A,No discount available

Examples by Use Case

Example 1: Product Catalog

ProductName,SKU,Category,Price,Description,InStock,ShippingWeight
Deluxe Widget,DLX-001,Widgets,299.99,"Premium widget with advanced features including X, Y, and Z",Yes,2.5 lbs
Standard Gadget,STD-002,Gadgets,149.99,"Reliable gadget for everyday use with features A and B",Yes,1.2 lbs
Pro Service Pack,PRO-100,Services,499.99,"Complete service package including setup, training, and 6-month support",Yes,N/A

Questions AI can answer:

• "What widgets do you have?"

• "How much is the Deluxe Widget?"

• "What's in stock?"

• "Show me products under $200"

• "What services do you offer?"

Example 2: Office Locations

OfficeName,Address,City,State,Zip,Country,Phone,Email,Hours,Specialization
Headquarters,123 Tech Boulevard,San Francisco,CA,94105,USA,555-0100,[email protected],Mon-Fri 8AM-6PM,"Sales, Support, R&D"
East Coast Office,456 Innovation Drive,Boston,MA,02108,USA,555-0200,[email protected],Mon-Fri 9AM-5PM,"Sales, Support"
European Hub,789 Tech Street,London,N/A,EC1A 1BB,UK,[phone],[email protected],Mon-Fri 9AM-5PM GMT,"Sales, Support, EU Operations"

Questions AI can answer:

• "Where are your offices located?"

• "What's the phone number for the Boston office?"

• "What are the hours for the London office?"

• "Which offices provide R&D services?"

Example 3: Pricing Plans

PlanName,MonthlyPrice,AnnualPrice,MaxUsers,StorageGB,Features,Support,BestFor
Starter,29.99,299.90,5,10,"Basic features, Email integration, Mobile app",Email only,"Small teams, Basic needs"
Professional,99.99,999.90,25,100,"All Starter features plus Advanced analytics, API access, Integrations, Custom branding",Email + Chat,"Growing teams, Professional use"
Enterprise,Custom,Custom,Unlimited,Unlimited,"All Professional features plus SSO, Advanced security, Dedicated account manager, Custom integrations, SLA",24/7 Phone + Dedicated support,"Large organizations, Complex requirements"

Questions AI can answer:

• "What plans do you offer?"

• "What's included in the Professional plan?"

• "How much does the Starter plan cost annually?"

• "Which plan is best for a small team?"

• "Do you offer SSO?"

Example 4: FAQ + Metadata

Question,Answer,Category,LastUpdated,RelatedProducts
How do I reset my password?,"Go to login page, click 'Forgot Password', and follow email instructions.",Account,2024-01-15,N/A
What's your return policy?,"30-day returns on unused items with original packaging.",Policies,2024-01-10,"All products"
How do I contact support?,"Email: [email protected], Phone: [phone], Live Chat on website",Support,2024-01-20,N/A

Best Practices

1. Data Organization

Denormalize for AI: Instead of:

ProductID,Name
1,Widget A
2,Widget B

and separate

ProductID,Price
1,99.99
2,149.99

Use:

ProductID,Name,Price,Description
1,Widget A,99.99,Full description...
2,Widget B,149.99,Full description...

2. Include Context in Columns

Add descriptive columns that provide context:

Product,Price,WhenToUse,WhoItsFor
Basic Plan,29.99,"For simple needs and getting started","Individuals and small teams up to 5 people"
Pro Plan,99.99,"When you need advanced features and more capacity","Growing businesses with 10-50 employees"

3. Use Full Text, Not Codes

Good:

Product,Status,Availability
Widget A,In Stock,Ships within 24 hours
Widget B,Low Stock,Limited quantities available
Widget C,Backordered,Ships in 2-3 weeks

Avoid:

Product,Status,Availability
Widget A,IS,24H
Widget B,LS,LQ
Widget C,BO,2-3W

4. Consistent Formatting

• Dates: Use ISO format (YYYY-MM-DD)

• Currency: Include symbols ($99.99 or 99.99 USD)

• Boolean: Use Yes/No, True/False, or 1/0 consistently

• Lists: Use semicolons or pipes to separate multiple values

Product,Price,Features,Available
Widget A,$99.99,"Feature 1; Feature 2; Feature 3",Yes
Widget B,$149.99,"Feature X; Feature Y; Feature Z",Yes

Technical Requirements

File Format

• Extension: .csv

• Encoding: UTF-8 (recommended)

• Delimiter: Comma (,)

• Line Endings: LF (\n) or CRLF (\r\n)

Size Limits

• Varies by plan

• Split large datasets into multiple CSVs if needed

• Consider sampling for very large datasets

Headers

• Must have header row with column names

• No duplicate column names

• Avoid special characters in column names

Updating Your Data CSV

To update your data:

1. Edit your local CSV file with changes

2. Navigate to the data source in Twig

3. Click Edit

4. Upload the updated CSV file

5. Click Save to reprocess

Note: The entire dataset is replaced, not merged.

Troubleshooting

Data Not Searchable

Problem: AI can't find information in the CSV

Solutions:

• Verify column headers are descriptive

• Add more descriptive text in columns

• Include context columns explaining the data

• Check encoding is UTF-8

Incorrect Data Interpretation

Problem: AI misinterprets the data

Solutions:

• Use full text instead of abbreviations

• Add descriptive column names

• Include units in data (e.g., "Price USD" instead of "Price")

• Add context columns

Processing Errors

Problem: CSV fails to process

Solutions:

• Check for malformed CSV (missing quotes, unescaped commas)

• Verify file encoding

• Look for special characters

• Ensure consistent column count across rows

Missing Rows

Problem: Some rows not imported

Solutions:

• Check for empty required fields

• Verify no hidden characters

• Ensure no duplicate headers

• Review process logs for specific errors

Advanced Use Cases

1. Multi-Language Support

ProductName,DescriptionEN,DescriptionES,DescriptionFR,Price
Widget A,"High-quality widget","Widget de alta calidad","Widget de haute qualité",99.99

2. Rich Product Information

Product,Description,Specifications,UseCases,Compatibility,Warranty
Premium Widget,"Detailed description...","Size: 10x10x5, Weight: 2lbs, Material: Aluminum","Perfect for X, Y, and Z scenarios","Works with Product A, B, and C","2-year limited warranty"

3. Hierarchical Data (Flattened)

Category,Subcategory,Product,Description,Price
Electronics,Computers,Laptop A,"13-inch laptop with...",999.99
Electronics,Computers,Laptop B,"15-inch laptop with...",1299.99
Electronics,Tablets,Tablet X,"10-inch tablet with...",499.99

Next Steps

After importing your Data CSV:

1. Test queries to verify AI can access the data

2. Create specialized agents for specific data types

3. Monitor usage analytics to see how data is queried

4. Refine and expand your CSV based on user questions

Related Connectors

• QnA CSV - Question-answer pairs

• Files - Upload document files

• Confluence - Structured wiki content

• Google Drive - Sync spreadsheets from Drive