Overview
AWS-based ikman-crawler data ingestor to Pinecone vector database. Developed AWS lambda triggers for DynamoDB and Lambda function to upload data into Pinecone Vector database.
This project was developed at Zero One Technologies to create a data pipeline from ikman.lk to a vector database.
Key components:
- AWS Lambda triggers for DynamoDB streams
- Data processing pipeline to convert text data to vector embeddings
- Integration with Pinecone Vector Database through API calls
- Automated monitoring and error handling
The system captures new entries from ikman.lk, processes them through an AI model to generate vector embeddings, and stores them in Pinecone for semantic search capabilities.
Project Articles
How to create an efficient serverless data pipeline using AWS Lambda, DynamoDB, and Pinecone for vector search capabilities.
How vector embeddings transform e-commerce search from keyword matching to semantic understanding, with implementation details from our ikman.lk project.