Commit Graph

51 Commits

Author SHA1 Message Date
Josako
51fd16bcc6 - RAG Specialist fully implemented new style
- Selection Specialist - VA version - fully implemented
- Correction of TRAICIE_ROLE_DEFINITION_SPECIALIST - adaptation to new style
- Removal of 'debug' statements
2025-07-10 10:39:42 +02:00
Josako
f5c9542a49 - Introducing translation service prompts
- Ensure Traicie Role Definition Specialist complies to latest technical requirements
- Ensure that empty historical messages do not cause a crash in eveai_client
- take into account empty customisation options
- make was not processed in the system dynamic attribute tenant_make
- ensure only relevant makes are shown when creating magic links
- refresh partner info when editing or adding Partner Services$
2025-06-24 14:15:36 +02:00
Josako
7b87880045 - Full Traicie Selection Specialist Flow implemented
- Added Specialist basics for handling phases and automatically transferring data between state and output
- Added QR-code generation for Magic Links
2025-06-23 11:46:56 +02:00
Josako
5b2c04501c - logging improvement and simplification (no more graylog)
- Traicie Selection Specialist Round Trip
- Session improvements + debugging enabled
- Tone of Voice & Langauge Level definitions introduced
2025-06-20 07:58:06 +02:00
Josako
1fdbd2ff45 - Move global config files to globals iso global folder, as the name global conflicts with python language
- Creation of Traicie Vancancy Definition specialist
- Allow to invoke non-interaction specialists from withing Evie's mgmt interface (eveai_app)
- Improvements to crewai specialized classes
- Introduction to json editor for showing specialists arguments and results in a better way
- Introduction of more complex pagination (adding extra arguments) by adding a global 'get_pagination_html'
- Allow follow-up of ChatSession / Specialist execution
- Improvement in logging of Specialists (but needs to be finished)
2025-05-26 11:26:03 +02:00
Josako
70de4c0328 Improve HTML Processing + Introduction of Processed File viewer 2025-05-19 17:18:16 +02:00
Josako
12a53ebc1c - Convert mail messaging from SMTP to Scaleway TEM mails 2025-05-10 10:49:15 +02:00
Josako
4c480c9baa - Corrections for setting up the test environment
- Correction of some bugs discovered
2025-05-08 14:15:06 +02:00
Josako
4bf12db142 - Significantly changed the PDF Processor to use Mistral's OCR model
- ensure very long chunks get split into smaller chunks
- ensure TrackedMistralAIEmbedding is batched if needed to ensure correct execution
- upgraded some of the packages to a higher version
2025-04-16 15:39:16 +02:00
Josako
9ad7c1aee9 Introduction of Partner Model, adding code to Tenant model 2025-04-03 14:13:56 +02:00
Josako
b6ee7182de - Adding Prometheus and grafana services in development
- Adding Prometheus metrics to the business events
- Ensure asynchronous behaviour of crewai specialists.
- Adapt Business events to working in mixed synchronous / asynchronous contexts
- Extend business events with specialist information
- Started adding a grafana dashboard (TBC)
2025-03-24 16:39:22 +01:00
Josako
238bdb58f4 - Upgrade to crewai v108 2025-03-18 14:46:06 +01:00
Josako
55a89c11bb - Move from OpenAI to Mistral Embeddings
- Move embedding model settings from tenant to catalog
- BUG: error processing configuration for chunking patterns in HTML_PROCESSOR
- Removed eveai_chat from docker-files and nginx configuration, as it is now obsolete
- BUG: error in Library Operations when creating a new default RAG library
- BUG: Added public type in migration scripts
- Removed SocketIO from all code and requirements.txt
2025-02-25 11:17:19 +01:00
Josako
25213f2004 - Implementation of specialist execution api, including SSE protocol
- eveai_chat becomes deprecated and should be replaced with SSE
- Adaptation of STANDARD_RAG specialist
- Base class definition allowing to realise specialists with crewai framework
- Implementation of SPIN_SPECIALIST
- Implementation of test app for testing specialists (test_specialist_client). Also serves as an example for future SSE-based client
- Improvements to startup scripts to better handle and scale multiple connections
- Small improvements to the interaction forms and views
- Caching implementation improved and augmented with additional caches
2025-02-20 05:50:16 +01:00
Josako
d106520d22 - Finish editing of Specialists with overview, agent - task - tool editor
- Split differrent caching mechanisms (types, version tree, config) into different cachers
- Improve resource usage on starting components, and correct gevent usage
- Refine repopack usage for eveai_app (too large)
- Change nginx dockerfile to allow for specialist overviews being served statically
2025-01-23 09:43:48 +01:00
Josako
d35ec9f5ae - Addition of general chunking parameters chunking_heading_level and chunking patterns
- Addition of Processor types docx and markdown
2024-12-05 15:19:37 +01:00
Josako
99135c9b02 - Updated OpenAI client due to 'proxies' error (no longer supported). 2024-11-29 14:11:07 +01:00
Josako
1807435339 - Introduction of dynamic Retrievers & Specialists
- Introduction of dynamic Processors
- Introduction of caching system
- Introduction of a better template manager
- Adaptation of ModelVariables to support dynamic Processors / Retrievers / Specialists
- Start adaptation of chat client
2024-11-15 10:00:53 +01:00
Josako
aa358df28e - Allowing for multiple types of Catalogs
- Introduction of retrievers
- Ensuring processing information is collected from Catalog iso Tenant
- Introduction of a generic Form class to enable dynamic fields based on a configuration
- Realisation of Retriever functionality to support dynamic fields
2024-10-25 14:11:47 +02:00
Josako
9f5f090f0c - License Usage Calculation realised
- View License Usages
- Celery Beat container added
- First schedule in Celery Beat for calculating usage (hourly)
- repopack can now split for different components
- Various fixes as consequece of changing file_location / file_name ==> bucket_name / object_name
- Celery Routing / Queuing updated
2024-10-11 16:33:36 +02:00
Josako
b700cfac64 - Improvements on audio processing to limit CPU and memory usage
- Removed Portkey from the equation, and defined explicit monitoring using Langchain native code
- Optimization of Business Event logging
2024-10-02 14:11:46 +02:00
Josako
883175b8f5 - Portkey log retrieval started
- flower container added (dev and prod)
2024-10-01 08:01:59 +02:00
Josako
d9cb00fcdc Business event tracing completed for both eveai_workers tasks and eveai_chat_workers tasks 2024-09-27 10:53:42 +02:00
Josako
a740c96630 - turned model_variables into a class with lazy loading
- some improvements to Healthchecks
2024-09-24 10:48:52 +02:00
Josako
6cf660e622 - Adding a Tenant Type
- Allow filtering on Tenant Types & searching for parts of Tenant names
- Implement health checks
- Start Prometheus monitoring (needs to be finalized)
- Refine audio_processor and srt_processor to reduce duplicate code and support for larger files
- Introduce repopack to reason in LLMs about the code
2024-09-13 15:43:40 +02:00
Josako
76cb825660 - Full API application, streamlined, de-duplication of document handling code into document_utils.py
- Added meta-data fields to DocumentVersion
- Docker container to support API
2024-09-09 16:11:42 +02:00
Josako
914c265afe - Improvements on document uploads (accept other files than html-files when entering a URL)
- Introduction of API-functionality (to be continued). Deduplication of document and url uploads between views and api.
- Improvements on document processing - introduction of processor classes to streamline document inputs
- Removed pure Youtube functionality, as Youtube retrieval of documents continuously changes. But added upload of srt, mp3, ogg and mp4
2024-09-02 12:37:44 +02:00
Josako
122d1a18df - Allow for more complex and longer PDFs to be uploaded to Evie. First implmentation of a processor for specific file types.
- Allow URLs to contain other information than just HTML information. It can alose refer to e.g. PDF-files.
2024-08-27 07:05:56 +02:00
Josako
2ca006d82c Added excluded element classes to HTML parsing to allow for more complex document parsing
Added chunking to conversion of HTML to markdown in case of large files
2024-08-22 16:41:13 +02:00
Josako
ab38dd7540 - Improvements working with the cloud, minio, graylog and some first bugfixing 2024-08-13 09:04:19 +02:00
Josako
9caa423bcf - Getting containers ready for the cloud 2024-08-07 08:37:13 +02:00
Josako
ab8359365d - Getting containers ready for the cloud 2024-08-06 14:56:12 +02:00
Josako
64cf8df3a9 - Improvements to enable deployment in the cloud, mainly changing file access to Minio
- Improvements on RAG logging, and some debugging in that area
2024-08-01 17:35:54 +02:00
Josako
88ca04136d Portkey Integration 2024-07-18 07:08:42 +02:00
Josako
908a2eaf7e - Improve annotation algorithm for Youtube (and others)
- Patch Pytube
- improve OS deletion of files and writing of files
- Start working on Claude
- Improve template management
2024-07-16 14:21:49 +02:00
Josako
ea0127b4b8 Improve algorithms for HTML and PDF processing 2024-07-08 15:20:45 +02:00
Josako
8e1dac0233 Youtube added - further checking required 2024-07-04 08:11:31 +02:00
Josako
7a1b51dd0c Docker deployment Realised 2024-06-26 12:02:14 +02:00
Josako
348bdf2c15 Commit before setting up a new deployment environment 2024-06-21 13:21:54 +02:00
Josako
fd510c8fcd Correcting the retrieval of relevant documents 2024-06-12 16:15:48 +02:00
Josako
be311c440b Improving chat functionality significantly throughout the application. 2024-06-12 11:07:18 +02:00
Josako
27b6de8734 Removing DocumentLanguage, as both System Context and User Context are to be defined on DocumentVersion level.
Finetuning of embedding workers.
2024-06-06 15:26:49 +02:00
Josako
fcc0caeb09 Optimizing admin interface for user domain, completing security views 2024-06-03 09:37:59 +02:00
Josako
ea23e8d327 Setting up the chat client functionality using SocketIO - Start 2024-05-15 14:37:21 +02:00
Josako
011bdce38d Prepare for html document validation (added wanted tags to tenant) 2024-05-12 21:58:42 +02:00
Josako
699de951e8 Add functionality to add a URL to the system. 2024-05-10 22:44:53 +02:00
Josako
d925477e68 Setup of documents view 2024-05-05 20:21:44 +02:00
Josako
0b7f565547 gevent concurrency / asynchronous processes
usage of gunicorn (documented)
2024-04-30 15:50:12 +02:00
Josako
ffa60b4616 Update Document domain models to use pgvector (extension of PostgreSQL) 2024-04-30 15:09:32 +02:00
Josako
f4decbc0cd More dynamic menu handling. Did a lot of stuff using Flask-Nav & Flask_Menu, but removed all of it becauses it became overly complex or the extensions were no longer in active development. 2024-04-28 22:15:47 +02:00