Files
plane/web/core/local-db/utils/indexes.ts

68 lines
2.2 KiB
TypeScript
Raw Normal View History

[WEB-2001]feat: Cache issues on the client (#5327) * use common getIssues from issue service instead of multiple different services for modules and cycles * Use SQLite to store issues locally and load issues from it. * Fix incorrect total count and filtering on assignees. * enable parallel API calls * use common getIssues from issue service instead of multiple different services for modules and cycles * Use SQLite to store issues locally and load issues from it. * Fix incorrect total count and filtering on assignees. * enable parallel API calls * chore: deleted issue list * - Handle local mutations - Implement getting the updates - Use SWR to update/sync data * Wait for sync to complete in get issues * Fix build errors * Fix build issue * - Sync updates to local-db - Fallback to server when the local data is loading - Wait when the updates are being fetched * Add issues in batches * Disable skeleton loaders for first 10 issues * Load issues in bulk * working version of sql lite with grouped issues * Use window queries for group by * - Fix sort by date fields - Fix the total count * - Fix grouping by created by - Fix order by and limit * fix pagination * Fix sorting on issue priority * - Add secondary sort order - Fix group by priority * chore: added timestamp filter for deleted issues * - Extract local DB into its own class - Implement sorting by label names * Implement subgroup by * sub group by changes * Refactor query constructor * Insert or update issues instead of directly adding them. * Segregated queries. Not working though!! * - Get filtered issues and then group them. - Cleanup code. - Implement order by labels. * Fix build issues * Remove debuggers * remove loaders while changing sorting or applying filters * fix loader while clearing all filters * Fix issue with project being synced twice * Improve project sync * Optimize the queries * Make create dummy data more realistic * dev: added total pages in the global paginator * chore: updated total_paged count * chore: added state_group in the issues pagination * chore: removed deleted_at from the issue pagination payload * chore: replaced state_group with state__group * Integrate new getIssues API, and fix sync issues bug. * Fix issue with SWR running twice in workspace wrapper * Fix DB initialization called when opening project for the first time. * Add all the tables required for sorting * Exclude description from getIssues * Add getIssue function. * Add only selected fields to get query. * Fix the count query * Minor query optimization when no joins are required. * fetch issue description from local db * clear local db on signout * Correct dummy data creation * Fix sort by assignee * sync to local changes * chore: added archived issues in the deleted endpoint * Sync deletes to local db. * - Add missing indexes for tables used in sorting in spreadsheet layout. - Add options table * Make fallback optional in getOption * Kanban column virtualization * persist project sync readiness to sqlite and use that as the source of truth for the project issues to be ready * fix build errors * Fix calendar view * fetch slimed down version of modules in project wrapper * fetch toned down modules and then fetch complete modules * Fix multi value order by in spread sheet layout * Fix sort by * Fix the query when ordering by multi field names * Remove unused import * Fix sort by multi value fields * Format queries and fix order by * fix order by for multi issue * fix loaders for spreadsheet * Fallback to manual order whn moving away from spreadsheet layout * fix minor bug * Move fix for order_by when switching from spreadsheet layout to translateQueryParams * fix default rendering of kanban groups * Fix none priority being saved as null * Remove debugger statement * Fix issue load * chore: updated isue paginated query from to * Fix sub issues and start and target date filters * Fix active and backlog filter * Add default order by * Update the Query param to match with backend. * local sqlite db versioning * When window is hidden, do not perform any db versioning * fix error handling and fall back to server when database errors out * Add ability to disable local db cache * remove db version check from getIssues function * change db version to number and remove workspaceInitPromise in storage.sqlite * - Sync the entire workspace in the background - Add get sub issue method with distribution * Make changes to get issues for sync to match backend. * chore: handled workspace and project in v2 paginted issues * disable issue description and title until fetched from server * sync issues post bulk operations * fix server error * fix front end build * Remove full workspace sync * - Remove the toast message on sync. - Update the disable local message. * Add Hardcoded constant to disable the local db caching * fix lint errors * Fix order by in grouping * update yarn lock * fix build * fix plane-web imports * address review comments --------- Co-authored-by: rahulramesha <rahulramesham@gmail.com> Co-authored-by: NarayanBavisetti <narayan3119@gmail.com> Co-authored-by: gurusainath <gurusainath007@gmail.com>
2024-09-24 19:01:34 +05:30
import { persistence } from "../storage.sqlite";
[WEB-2706] chore: Switch to wa-sqlite (#5859) * fix layout switching when filter is not yet completely fetched * add layout in issue filter params * Handle cases when DB intilization failed * chore: permission layer and updated issues v1 query from workspace to project level * - Switch to using wa-sqlite instead of sqlite-wasm * Code cleanup and fix indexes * Add missing files * - Import only required functions from sentry - Wait till all the tables are created * Skip workspace sync if one is already in progress. * Sync workspace without using transaction * Minor cleanup * Close DB connection before deleting files Fix clear OPFS on safari * Fix type issue * Improve issue insert performance * Refactor workspace sync * Close the DB connection while switching workspaces * Update web/core/local-db/worker/db.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Worker cleanup and error handling Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update web/core/local-db/worker/db.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update web/core/local-db/storage.sqlite.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update web/core/local-db/worker/db.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Code cleanup * Set default order by to created at and descending * Wait for transactions to complete. --------- Co-authored-by: rahulramesha <rahulramesham@gmail.com> Co-authored-by: gurusainath <gurusainath007@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2024-10-24 15:35:02 +05:30
import { log } from "./utils";
[WEB-2001]feat: Cache issues on the client (#5327) * use common getIssues from issue service instead of multiple different services for modules and cycles * Use SQLite to store issues locally and load issues from it. * Fix incorrect total count and filtering on assignees. * enable parallel API calls * use common getIssues from issue service instead of multiple different services for modules and cycles * Use SQLite to store issues locally and load issues from it. * Fix incorrect total count and filtering on assignees. * enable parallel API calls * chore: deleted issue list * - Handle local mutations - Implement getting the updates - Use SWR to update/sync data * Wait for sync to complete in get issues * Fix build errors * Fix build issue * - Sync updates to local-db - Fallback to server when the local data is loading - Wait when the updates are being fetched * Add issues in batches * Disable skeleton loaders for first 10 issues * Load issues in bulk * working version of sql lite with grouped issues * Use window queries for group by * - Fix sort by date fields - Fix the total count * - Fix grouping by created by - Fix order by and limit * fix pagination * Fix sorting on issue priority * - Add secondary sort order - Fix group by priority * chore: added timestamp filter for deleted issues * - Extract local DB into its own class - Implement sorting by label names * Implement subgroup by * sub group by changes * Refactor query constructor * Insert or update issues instead of directly adding them. * Segregated queries. Not working though!! * - Get filtered issues and then group them. - Cleanup code. - Implement order by labels. * Fix build issues * Remove debuggers * remove loaders while changing sorting or applying filters * fix loader while clearing all filters * Fix issue with project being synced twice * Improve project sync * Optimize the queries * Make create dummy data more realistic * dev: added total pages in the global paginator * chore: updated total_paged count * chore: added state_group in the issues pagination * chore: removed deleted_at from the issue pagination payload * chore: replaced state_group with state__group * Integrate new getIssues API, and fix sync issues bug. * Fix issue with SWR running twice in workspace wrapper * Fix DB initialization called when opening project for the first time. * Add all the tables required for sorting * Exclude description from getIssues * Add getIssue function. * Add only selected fields to get query. * Fix the count query * Minor query optimization when no joins are required. * fetch issue description from local db * clear local db on signout * Correct dummy data creation * Fix sort by assignee * sync to local changes * chore: added archived issues in the deleted endpoint * Sync deletes to local db. * - Add missing indexes for tables used in sorting in spreadsheet layout. - Add options table * Make fallback optional in getOption * Kanban column virtualization * persist project sync readiness to sqlite and use that as the source of truth for the project issues to be ready * fix build errors * Fix calendar view * fetch slimed down version of modules in project wrapper * fetch toned down modules and then fetch complete modules * Fix multi value order by in spread sheet layout * Fix sort by * Fix the query when ordering by multi field names * Remove unused import * Fix sort by multi value fields * Format queries and fix order by * fix order by for multi issue * fix loaders for spreadsheet * Fallback to manual order whn moving away from spreadsheet layout * fix minor bug * Move fix for order_by when switching from spreadsheet layout to translateQueryParams * fix default rendering of kanban groups * Fix none priority being saved as null * Remove debugger statement * Fix issue load * chore: updated isue paginated query from to * Fix sub issues and start and target date filters * Fix active and backlog filter * Add default order by * Update the Query param to match with backend. * local sqlite db versioning * When window is hidden, do not perform any db versioning * fix error handling and fall back to server when database errors out * Add ability to disable local db cache * remove db version check from getIssues function * change db version to number and remove workspaceInitPromise in storage.sqlite * - Sync the entire workspace in the background - Add get sub issue method with distribution * Make changes to get issues for sync to match backend. * chore: handled workspace and project in v2 paginted issues * disable issue description and title until fetched from server * sync issues post bulk operations * fix server error * fix front end build * Remove full workspace sync * - Remove the toast message on sync. - Update the disable local message. * Add Hardcoded constant to disable the local db caching * fix lint errors * Fix order by in grouping * update yarn lock * fix build * fix plane-web imports * address review comments --------- Co-authored-by: rahulramesha <rahulramesham@gmail.com> Co-authored-by: NarayanBavisetti <narayan3119@gmail.com> Co-authored-by: gurusainath <gurusainath007@gmail.com>
2024-09-24 19:01:34 +05:30
export const createIssueIndexes = async () => {
const columns = [
"state_id",
"sort_order",
// "priority",
"priority_proxy",
"project_id",
"created_by",
"cycle_id",
"sequence_id",
[WEB-2001]feat: Cache issues on the client (#5327) * use common getIssues from issue service instead of multiple different services for modules and cycles * Use SQLite to store issues locally and load issues from it. * Fix incorrect total count and filtering on assignees. * enable parallel API calls * use common getIssues from issue service instead of multiple different services for modules and cycles * Use SQLite to store issues locally and load issues from it. * Fix incorrect total count and filtering on assignees. * enable parallel API calls * chore: deleted issue list * - Handle local mutations - Implement getting the updates - Use SWR to update/sync data * Wait for sync to complete in get issues * Fix build errors * Fix build issue * - Sync updates to local-db - Fallback to server when the local data is loading - Wait when the updates are being fetched * Add issues in batches * Disable skeleton loaders for first 10 issues * Load issues in bulk * working version of sql lite with grouped issues * Use window queries for group by * - Fix sort by date fields - Fix the total count * - Fix grouping by created by - Fix order by and limit * fix pagination * Fix sorting on issue priority * - Add secondary sort order - Fix group by priority * chore: added timestamp filter for deleted issues * - Extract local DB into its own class - Implement sorting by label names * Implement subgroup by * sub group by changes * Refactor query constructor * Insert or update issues instead of directly adding them. * Segregated queries. Not working though!! * - Get filtered issues and then group them. - Cleanup code. - Implement order by labels. * Fix build issues * Remove debuggers * remove loaders while changing sorting or applying filters * fix loader while clearing all filters * Fix issue with project being synced twice * Improve project sync * Optimize the queries * Make create dummy data more realistic * dev: added total pages in the global paginator * chore: updated total_paged count * chore: added state_group in the issues pagination * chore: removed deleted_at from the issue pagination payload * chore: replaced state_group with state__group * Integrate new getIssues API, and fix sync issues bug. * Fix issue with SWR running twice in workspace wrapper * Fix DB initialization called when opening project for the first time. * Add all the tables required for sorting * Exclude description from getIssues * Add getIssue function. * Add only selected fields to get query. * Fix the count query * Minor query optimization when no joins are required. * fetch issue description from local db * clear local db on signout * Correct dummy data creation * Fix sort by assignee * sync to local changes * chore: added archived issues in the deleted endpoint * Sync deletes to local db. * - Add missing indexes for tables used in sorting in spreadsheet layout. - Add options table * Make fallback optional in getOption * Kanban column virtualization * persist project sync readiness to sqlite and use that as the source of truth for the project issues to be ready * fix build errors * Fix calendar view * fetch slimed down version of modules in project wrapper * fetch toned down modules and then fetch complete modules * Fix multi value order by in spread sheet layout * Fix sort by * Fix the query when ordering by multi field names * Remove unused import * Fix sort by multi value fields * Format queries and fix order by * fix order by for multi issue * fix loaders for spreadsheet * Fallback to manual order whn moving away from spreadsheet layout * fix minor bug * Move fix for order_by when switching from spreadsheet layout to translateQueryParams * fix default rendering of kanban groups * Fix none priority being saved as null * Remove debugger statement * Fix issue load * chore: updated isue paginated query from to * Fix sub issues and start and target date filters * Fix active and backlog filter * Add default order by * Update the Query param to match with backend. * local sqlite db versioning * When window is hidden, do not perform any db versioning * fix error handling and fall back to server when database errors out * Add ability to disable local db cache * remove db version check from getIssues function * change db version to number and remove workspaceInitPromise in storage.sqlite * - Sync the entire workspace in the background - Add get sub issue method with distribution * Make changes to get issues for sync to match backend. * chore: handled workspace and project in v2 paginted issues * disable issue description and title until fetched from server * sync issues post bulk operations * fix server error * fix front end build * Remove full workspace sync * - Remove the toast message on sync. - Update the disable local message. * Add Hardcoded constant to disable the local db caching * fix lint errors * Fix order by in grouping * update yarn lock * fix build * fix plane-web imports * address review comments --------- Co-authored-by: rahulramesha <rahulramesham@gmail.com> Co-authored-by: NarayanBavisetti <narayan3119@gmail.com> Co-authored-by: gurusainath <gurusainath007@gmail.com>
2024-09-24 19:01:34 +05:30
];
const promises: Promise<any>[] = [];
promises.push(persistence.db.exec({ sql: `CREATE UNIQUE INDEX issues_issue_id_idx ON issues (id)` }));
columns.forEach((column) => {
promises.push(
persistence.db.exec({ sql: `CREATE INDEX issues_issue_${column}_idx ON issues (project_id, ${column})` })
);
});
await Promise.all(promises);
};
export const createIssueMetaIndexes = async () => {
// Drop indexes
await persistence.db.exec({ sql: `CREATE INDEX issue_meta_all_idx ON issue_meta (issue_id,key,value)` });
};
export const createWorkSpaceIndexes = async () => {
const promises: Promise<any>[] = [];
// Labels
promises.push(persistence.db.exec({ sql: `CREATE INDEX labels_name_idx ON labels (id,name,project_id)` }));
// Modules
promises.push(persistence.db.exec({ sql: `CREATE INDEX modules_name_idx ON modules (id,name,project_id)` }));
// States
promises.push(persistence.db.exec({ sql: `CREATE INDEX states_name_idx ON states (id,name,project_id)` }));
// Cycles
promises.push(persistence.db.exec({ sql: `CREATE INDEX cycles_name_idx ON cycles (id,name,project_id)` }));
// Members
promises.push(persistence.db.exec({ sql: `CREATE INDEX members_name_idx ON members (id,first_name)` }));
// Estimate Points @todo
promises.push(persistence.db.exec({ sql: `CREATE INDEX estimate_points_name_idx ON estimate_points (id,value)` }));
// Options
[WEB-2706] chore: Switch to wa-sqlite (#5859) * fix layout switching when filter is not yet completely fetched * add layout in issue filter params * Handle cases when DB intilization failed * chore: permission layer and updated issues v1 query from workspace to project level * - Switch to using wa-sqlite instead of sqlite-wasm * Code cleanup and fix indexes * Add missing files * - Import only required functions from sentry - Wait till all the tables are created * Skip workspace sync if one is already in progress. * Sync workspace without using transaction * Minor cleanup * Close DB connection before deleting files Fix clear OPFS on safari * Fix type issue * Improve issue insert performance * Refactor workspace sync * Close the DB connection while switching workspaces * Update web/core/local-db/worker/db.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Worker cleanup and error handling Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update web/core/local-db/worker/db.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update web/core/local-db/storage.sqlite.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update web/core/local-db/worker/db.ts Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Code cleanup * Set default order by to created at and descending * Wait for transactions to complete. --------- Co-authored-by: rahulramesha <rahulramesham@gmail.com> Co-authored-by: gurusainath <gurusainath007@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2024-10-24 15:35:02 +05:30
promises.push(persistence.db.exec({ sql: `CREATE INDEX options_key_idx ON options (key)` }));
[WEB-2001]feat: Cache issues on the client (#5327) * use common getIssues from issue service instead of multiple different services for modules and cycles * Use SQLite to store issues locally and load issues from it. * Fix incorrect total count and filtering on assignees. * enable parallel API calls * use common getIssues from issue service instead of multiple different services for modules and cycles * Use SQLite to store issues locally and load issues from it. * Fix incorrect total count and filtering on assignees. * enable parallel API calls * chore: deleted issue list * - Handle local mutations - Implement getting the updates - Use SWR to update/sync data * Wait for sync to complete in get issues * Fix build errors * Fix build issue * - Sync updates to local-db - Fallback to server when the local data is loading - Wait when the updates are being fetched * Add issues in batches * Disable skeleton loaders for first 10 issues * Load issues in bulk * working version of sql lite with grouped issues * Use window queries for group by * - Fix sort by date fields - Fix the total count * - Fix grouping by created by - Fix order by and limit * fix pagination * Fix sorting on issue priority * - Add secondary sort order - Fix group by priority * chore: added timestamp filter for deleted issues * - Extract local DB into its own class - Implement sorting by label names * Implement subgroup by * sub group by changes * Refactor query constructor * Insert or update issues instead of directly adding them. * Segregated queries. Not working though!! * - Get filtered issues and then group them. - Cleanup code. - Implement order by labels. * Fix build issues * Remove debuggers * remove loaders while changing sorting or applying filters * fix loader while clearing all filters * Fix issue with project being synced twice * Improve project sync * Optimize the queries * Make create dummy data more realistic * dev: added total pages in the global paginator * chore: updated total_paged count * chore: added state_group in the issues pagination * chore: removed deleted_at from the issue pagination payload * chore: replaced state_group with state__group * Integrate new getIssues API, and fix sync issues bug. * Fix issue with SWR running twice in workspace wrapper * Fix DB initialization called when opening project for the first time. * Add all the tables required for sorting * Exclude description from getIssues * Add getIssue function. * Add only selected fields to get query. * Fix the count query * Minor query optimization when no joins are required. * fetch issue description from local db * clear local db on signout * Correct dummy data creation * Fix sort by assignee * sync to local changes * chore: added archived issues in the deleted endpoint * Sync deletes to local db. * - Add missing indexes for tables used in sorting in spreadsheet layout. - Add options table * Make fallback optional in getOption * Kanban column virtualization * persist project sync readiness to sqlite and use that as the source of truth for the project issues to be ready * fix build errors * Fix calendar view * fetch slimed down version of modules in project wrapper * fetch toned down modules and then fetch complete modules * Fix multi value order by in spread sheet layout * Fix sort by * Fix the query when ordering by multi field names * Remove unused import * Fix sort by multi value fields * Format queries and fix order by * fix order by for multi issue * fix loaders for spreadsheet * Fallback to manual order whn moving away from spreadsheet layout * fix minor bug * Move fix for order_by when switching from spreadsheet layout to translateQueryParams * fix default rendering of kanban groups * Fix none priority being saved as null * Remove debugger statement * Fix issue load * chore: updated isue paginated query from to * Fix sub issues and start and target date filters * Fix active and backlog filter * Add default order by * Update the Query param to match with backend. * local sqlite db versioning * When window is hidden, do not perform any db versioning * fix error handling and fall back to server when database errors out * Add ability to disable local db cache * remove db version check from getIssues function * change db version to number and remove workspaceInitPromise in storage.sqlite * - Sync the entire workspace in the background - Add get sub issue method with distribution * Make changes to get issues for sync to match backend. * chore: handled workspace and project in v2 paginted issues * disable issue description and title until fetched from server * sync issues post bulk operations * fix server error * fix front end build * Remove full workspace sync * - Remove the toast message on sync. - Update the disable local message. * Add Hardcoded constant to disable the local db caching * fix lint errors * Fix order by in grouping * update yarn lock * fix build * fix plane-web imports * address review comments --------- Co-authored-by: rahulramesha <rahulramesham@gmail.com> Co-authored-by: NarayanBavisetti <narayan3119@gmail.com> Co-authored-by: gurusainath <gurusainath007@gmail.com>
2024-09-24 19:01:34 +05:30
await Promise.all(promises);
};
const createIndexes = async () => {
log("### Creating indexes");
const start = performance.now();
const promises = [createIssueIndexes(), createIssueMetaIndexes(), createWorkSpaceIndexes()];
try {
await Promise.all(promises);
} catch (e) {
console.log((e as Error).message);
}
log("### Indexes created in", `${performance.now() - start}ms`);
};
export default createIndexes;