Project

General

Profile

Feature #1786

Consume events and sync data with cache

Added by Tab Tu 10 months ago. Updated 10 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Jennifer Jiang
Start date:
2024-06-22
Due date:
2024-09-22
% Done:

100%

Estimated time:
(Total: 0.00 h)

Description

We should focus on the 2 categories listed below:

listing data -> property_id(MLS) in cache -> partitioning by date in db
  • When extracting/downloading listing data from VOW/DDF/IDX or any data source, we should standardize from input. Columns on ORM should be structured in cache, and any other feed should be saved as a JSON doc via CLOB/BLOB type.
  • Create partitioning on loading date for all properties. Searching on the cache or database will always be accompanied by a date range to have a better performance.
  • Cache should be updated by the downloading schedule. e.g. VOW per 2 hours, DDF by daily, IDX by daily...
customer trainning data -> topics in cache -> vector result in db
  • LLM based data can be saved into different database. (PostgreSQL)
  • A DWH layer should be involved.
  • Create cache topics by format: custom_id + custom_topic + training_date.
  • Outdated topics should be removed from cache.

Subtasks

Feature #1807: Setup cache refresh scheduleClosedJennifer Jiang2024-07-212024-07-31

Actions
Feature #1808: Ingest customer trainning database (LLM)ClosedJennifer Jiang2024-07-282024-09-22

Actions
Feature #1809: Standardlize keys for listing input for all sourcesClosedJennifer Jiang2024-06-222024-07-30

Actions
Feature #1810: Setup cache schedule and health monitorClosedJennifer Jiang2024-08-052024-08-11

Actions
Feature #1811: Link centre DWH layer to redis cacheClosedJennifer Jiang2024-08-022024-08-14

Actions
Feature #1812: Stop ORMClosedJennifer Jiang2024-08-312024-09-10

Actions

No data to display

Also available in: Atom PDF