Commit Graph

71 Commits

Author SHA1 Message Date
Josako
cf2201a1f7 - Started addition of Assets (to e.g. handle document templates).
- To be continued (Models, first views are ready)
2025-03-17 17:40:42 +01:00
Josako
a6402524ce - Correct bug where URL can be too long due to tracking parameters ==> added clean_url function, to be called before adding an URL. 2025-03-17 17:39:32 +01:00
Josako
56a00c2894 - Add DOSSIER Catalog management possibilities to eveai_app. 2025-03-12 11:25:48 +01:00
Josako
5bfd3445bb - adding usage to specialist execution
- Correcting implementation of usage
- Removed some obsolete debug statements
2025-03-07 11:10:28 +01:00
Josako
c15cabc289 - Move to Mistral iso OpenAI as primary choice 2025-03-06 14:19:35 +01:00
Josako
55a89c11bb - Move from OpenAI to Mistral Embeddings
- Move embedding model settings from tenant to catalog
- BUG: error processing configuration for chunking patterns in HTML_PROCESSOR
- Removed eveai_chat from docker-files and nginx configuration, as it is now obsolete
- BUG: error in Library Operations when creating a new default RAG library
- BUG: Added public type in migration scripts
- Removed SocketIO from all code and requirements.txt
2025-02-25 11:17:19 +01:00
Josako
25213f2004 - Implementation of specialist execution api, including SSE protocol
- eveai_chat becomes deprecated and should be replaced with SSE
- Adaptation of STANDARD_RAG specialist
- Base class definition allowing to realise specialists with crewai framework
- Implementation of SPIN_SPECIALIST
- Implementation of test app for testing specialists (test_specialist_client). Also serves as an example for future SSE-based client
- Improvements to startup scripts to better handle and scale multiple connections
- Small improvements to the interaction forms and views
- Caching implementation improved and augmented with additional caches
2025-02-20 05:50:16 +01:00
Josako
d106520d22 - Finish editing of Specialists with overview, agent - task - tool editor
- Split differrent caching mechanisms (types, version tree, config) into different cachers
- Improve resource usage on starting components, and correct gevent usage
- Refine repopack usage for eveai_app (too large)
- Change nginx dockerfile to allow for specialist overviews being served statically
2025-01-23 09:43:48 +01:00
Josako
7bddeb0ebd - Add configuration of agents, tasks, tools, specialist in context of SPIN specialist
- correct startup of applications using gevent
- introduce startup scripts (eveai_app)
- caching manager for all configurations
2025-01-16 20:27:00 +01:00
Josako
f7cd58ed2a - Zapier Document Refresh action (create) added 2024-12-17 16:40:21 +01:00
Josako
46c60b36a0 - First 'working' version of the Zapier plugin. Needs further debugging and needs additional functionalty (only add_document.js) 2024-12-12 16:36:41 +01:00
Josako
d35ec9f5ae - Addition of general chunking parameters chunking_heading_level and chunking patterns
- Addition of Processor types docx and markdown
2024-12-05 15:19:37 +01:00
Josako
07d89d204f - Created a new eveai_chat plugin to support the new dynamic possibilities of the Specialists. Currently only supports standard Rag retrievers (i.e. no extra arguments). 2024-11-26 13:35:29 +01:00
Josako
7702a6dfcc - Modernized authentication with the introduction of TenantProject
- Created a base mail template
- Adapt and improve document API to usage of catalogs and processors
- Adapt eveai_sync to new authentication mechanism and usage of catalogs and processors
2024-11-21 17:24:33 +01:00
Josako
1807435339 - Introduction of dynamic Retrievers & Specialists
- Introduction of dynamic Processors
- Introduction of caching system
- Introduction of a better template manager
- Adaptation of ModelVariables to support dynamic Processors / Retrievers / Specialists
- Start adaptation of chat client
2024-11-15 10:00:53 +01:00
Josako
532073d38e - Add dynamic fields to DocumentVersion in case the Catalog requires it. 2024-10-30 13:52:18 +01:00
Josako
74cc7ae95e - Adapt Sync Wordpress Component to Catalog introduction
- Small bug fixes
2024-10-17 10:31:13 +02:00
Josako
6069f5f7e5 - Catalog functionality integrated into document and document_version views
- small bugfixes and improvements
2024-10-16 13:09:19 +02:00
Josako
3e644f1652 - Add Catalog Functionality 2024-10-15 18:14:57 +02:00
Josako
270479c77d - Add Catalog Concept to Document Domain
- Create Catalog views
- Modify document stack creation
2024-10-14 13:56:23 +02:00
Josako
9f5f090f0c - License Usage Calculation realised
- View License Usages
- Celery Beat container added
- First schedule in Celery Beat for calculating usage (hourly)
- repopack can now split for different components
- Various fixes as consequece of changing file_location / file_name ==> bucket_name / object_name
- Celery Routing / Queuing updated
2024-10-11 16:33:36 +02:00
Josako
d6a7743f26 - Minor corrections to entitlement changes and upgrades
- started new eveai_entitlements component (not finished)
2024-10-08 09:12:16 +02:00
Josako
9782e31ae5 - Refined entitlements to work with MiB for both embeddings and storage
- Improved DocumentVersion storage attributes to reflect Minio settings
- Added size to DocumentVersions to easily calculate usage
- License / LicenseTier forms and views added
2024-10-07 14:17:44 +02:00
Josako
b700cfac64 - Improvements on audio processing to limit CPU and memory usage
- Removed Portkey from the equation, and defined explicit monitoring using Langchain native code
- Optimization of Business Event logging
2024-10-02 14:11:46 +02:00
Josako
883175b8f5 - Portkey log retrieval started
- flower container added (dev and prod)
2024-10-01 08:01:59 +02:00
Josako
d9cb00fcdc Business event tracing completed for both eveai_workers tasks and eveai_chat_workers tasks 2024-09-27 10:53:42 +02:00
Josako
ee1b0f1cfa Start log tracing to log business events. Storage in both database and logging-backend. 2024-09-25 15:39:25 +02:00
Josako
a740c96630 - turned model_variables into a class with lazy loading
- some improvements to Healthchecks
2024-09-24 10:48:52 +02:00
Josako
67bdeac434 - Improvements and bugfixes to HealthChecks 2024-09-16 16:17:54 +02:00
Josako
6cf660e622 - Adding a Tenant Type
- Allow filtering on Tenant Types & searching for parts of Tenant names
- Implement health checks
- Start Prometheus monitoring (needs to be finalized)
- Refine audio_processor and srt_processor to reduce duplicate code and support for larger files
- Introduce repopack to reason in LLMs about the code
2024-09-13 15:43:40 +02:00
Josako
9e14824249 - Furter refinement of the API, adding functionality for refreshing documents and returning Token expiration time when retrieving token
- Implementation of a first version of a Wordpress plugin
- Adding api service to nginx.conf
2024-09-11 16:31:13 +02:00
Josako
76cb825660 - Full API application, streamlined, de-duplication of document handling code into document_utils.py
- Added meta-data fields to DocumentVersion
- Docker container to support API
2024-09-09 16:11:42 +02:00
Josako
341ba47d1c - Bugfixing 2024-09-05 14:31:54 +02:00
Josako
914c265afe - Improvements on document uploads (accept other files than html-files when entering a URL)
- Introduction of API-functionality (to be continued). Deduplication of document and url uploads between views and api.
- Improvements on document processing - introduction of processor classes to streamline document inputs
- Removed pure Youtube functionality, as Youtube retrieval of documents continuously changes. But added upload of srt, mp3, ogg and mp4
2024-09-02 12:37:44 +02:00
Josako
122d1a18df - Allow for more complex and longer PDFs to be uploaded to Evie. First implmentation of a processor for specific file types.
- Allow URLs to contain other information than just HTML information. It can alose refer to e.g. PDF-files.
2024-08-27 07:05:56 +02:00
Josako
2ca006d82c Added excluded element classes to HTML parsing to allow for more complex document parsing
Added chunking to conversion of HTML to markdown in case of large files
2024-08-22 16:41:13 +02:00
Josako
9757830bc4 Correct functions for creating new users, confirming email, resetting password and forgot password. 2024-08-21 14:59:56 +02:00
Josako
6219d11e56 Add a DocumentVersion overview that can be sorted and can be filtered. 2024-08-20 14:18:07 +02:00
Josako
926a4e8cc2 - Add support for environment in communication with portkey 2024-08-19 12:06:10 +02:00
Josako
912cb74bb7 Merge develop into main 2024-08-19 08:03:20 +02:00
Josako
a237db339a - Improved CSRF handling
- Wordpress plugin for Evie Chat
2024-08-13 14:31:29 +02:00
Josako
ab38dd7540 - Improvements working with the cloud, minio, graylog and some first bugfixing 2024-08-13 09:04:19 +02:00
Josako
ab8359365d - Getting containers ready for the cloud 2024-08-06 14:56:12 +02:00
Josako
64cf8df3a9 - Improvements to enable deployment in the cloud, mainly changing file access to Minio
- Improvements on RAG logging, and some debugging in that area
2024-08-01 17:35:54 +02:00
Josako
88ca04136d Portkey Integration 2024-07-18 07:08:42 +02:00
Josako
908a2eaf7e - Improve annotation algorithm for Youtube (and others)
- Patch Pytube
- improve OS deletion of files and writing of files
- Start working on Claude
- Improve template management
2024-07-16 14:21:49 +02:00
Josako
ea0127b4b8 Improve algorithms for HTML and PDF processing 2024-07-08 15:20:45 +02:00
Josako
c55fc6b7ce Added OpenAI large embedding / improve migration and upgrade handling for containers 2024-07-04 17:20:02 +02:00
Josako
8e1dac0233 Youtube added - further checking required 2024-07-04 08:11:31 +02:00
Josako
9187947f68 corrected container and other errors 2024-06-28 14:40:13 +02:00