Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Complete migration to CDP #539

Merged
merged 82 commits into from
May 29, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
a6cde09
Shift service worker to use CDP completely.
amovar18 Feb 27, 2024
dbcc23c
Fix JS cookies for CDP.
amovar18 Feb 27, 2024
0217951
Fix JS cookies.
amovar18 Feb 27, 2024
5067aff
Remove optional chaining for frame id
amovar18 Feb 27, 2024
1a692e9
Fix condition for frame filtering.
amovar18 Feb 28, 2024
9cea675
Fix frameId association.
amovar18 Feb 29, 2024
32e8ba8
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 Mar 1, 2024
9cf229b
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 Mar 1, 2024
b3715fb
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 Mar 26, 2024
3164869
Fix deletion of data at correct event.
amovar18 Mar 26, 2024
ab77f08
Get all cookies for current tab via CDP.
amovar18 Mar 27, 2024
9905de5
Add multitab support.
amovar18 Mar 28, 2024
1b5aae0
Add tab data when extension is updated.
amovar18 Mar 28, 2024
17b162e
Make multitab work
amovar18 Mar 29, 2024
684b8a6
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 Mar 29, 2024
1cfc4ad
Fix multi-tab analysis methodology for CDP.
amovar18 Mar 29, 2024
dda7c25
Keep previous frame.
amovar18 Mar 29, 2024
c66d6e6
Fix the inconsistency in frame and cookie association.
amovar18 Apr 1, 2024
f9aa67b
Fix removal of frameIdUrlSet.
amovar18 Apr 1, 2024
5e6f804
Add some conditions.
amovar18 Apr 4, 2024
514d4f3
Use loaderid to load other cookies.
amovar18 Apr 8, 2024
3cbda3d
Fix frame mapping of cookies.
amovar18 Apr 8, 2024
9b6d72e
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 Apr 8, 2024
ac5282d
Test fix.
amovar18 Apr 8, 2024
cec0c1d
revert test fix
amovar18 Apr 8, 2024
e060dfe
Destructure code to make it pretty.
amovar18 Apr 9, 2024
0ecfa13
Enable CDP by default
amovar18 Apr 9, 2024
341cb5d
Refactor code to make it pretty.
amovar18 Apr 9, 2024
0a900a7
Fix failing tests.
amovar18 Apr 9, 2024
ca5f652
Revert some changes
amovar18 Apr 9, 2024
84b45a4
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 Apr 10, 2024
7d3515b
Revert using loader id.
amovar18 Apr 10, 2024
3b92595
More precise mapping of data.
amovar18 Apr 10, 2024
9ed2c5f
Collect resource urls.
amovar18 Apr 11, 2024
2e5c2a3
collect frame resource urls.
amovar18 Apr 11, 2024
7a69eaa
Send cookies from resources.
amovar18 Apr 11, 2024
c15d4a3
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 Apr 11, 2024
adcdc39
Reinsert adding extra frame urls from service worker.
amovar18 Apr 11, 2024
73d33ac
Add comments explaining each thing in the chrome.debugger.
amovar18 Apr 11, 2024
a718b65
Fix some redundant issues remove auditsIssuesStore.
amovar18 Apr 11, 2024
fd04c4a
Fix errors thrown on extension page.
amovar18 Apr 12, 2024
be8c6c2
Add proper frameIds to map the cookies to frames better.
amovar18 Apr 12, 2024
eb86fd8
- Remove parentChildFrameAssociation from service worker.
amovar18 Apr 12, 2024
8b83745
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 Apr 18, 2024
7481b67
Fix some variable additions.
amovar18 Apr 18, 2024
a923e3e
Move all listeners to separate files.
amovar18 Apr 18, 2024
b2aa6a4
Split serviceworker into more listeners.
amovar18 Apr 18, 2024
b01ed48
Fix bug.
amovar18 Apr 18, 2024
64dfbaf
Fix failing tests.
amovar18 Apr 18, 2024
6463e78
Reintroduce webRequest API.
amovar18 Apr 22, 2024
324e914
Fix cli tsc problems.
amovar18 Apr 22, 2024
233bea0
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 Apr 22, 2024
f603733
create separate functions for listeners.
amovar18 Apr 23, 2024
ebd8bfb
Bug fix frame id association.
amovar18 Apr 23, 2024
aad04fc
Fix frame management after extension update.
amovar18 Apr 25, 2024
9f19a93
Add service worker test.
amovar18 Apr 26, 2024
076148a
Revert jest config.
amovar18 Apr 26, 2024
dd5d9be
Revert parsing method.
amovar18 Apr 29, 2024
f019676
Remove use of mainFrameId.
amovar18 Apr 29, 2024
1938f4d
Rearrange functions in sync cookie store.
amovar18 Apr 29, 2024
7009cac
Refactor code.
amovar18 Apr 29, 2024
a670de8
update frameId logic
amovar18 Apr 29, 2024
b8fa697
Revert logic of frame addition.
amovar18 Apr 29, 2024
579cd23
Refactor code and fix js cookie.
amovar18 Apr 30, 2024
34439bc
Move cookieDB to synchnorous cookie store.
amovar18 Apr 30, 2024
e71ff22
Fix arguments in getAndParseNetworkCookies
amovar18 Apr 30, 2024
50d1b1f
Fix failing tests.
amovar18 Apr 30, 2024
7f23040
Make non cdp environement work.
amovar18 May 1, 2024
63f23a6
Fix failing tests.
amovar18 May 1, 2024
efe88e0
Detach debugger from targets when cdp is switched off
amovar18 May 1, 2024
64b6be7
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 May 1, 2024
1bcacbe
Revert setting cdp as default true.
amovar18 May 1, 2024
f6edb0b
Remove exclude from exclusion reason.
amovar18 May 1, 2024
390fb6d
Delete cdpURLMapping.
amovar18 May 1, 2024
b0184f1
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
amovar18 May 17, 2024
9576087
Add jsdoc.
amovar18 May 22, 2024
eb22bc7
Use optional chaining.
amovar18 May 22, 2024
24da1cc
Fix jsdoc comment in index.ts
amovar18 May 22, 2024
1d2d359
Merge branch 'develop' of github.com:GoogleChromeLabs/ps-analysis-too…
mohdsayed May 28, 2024
0ecb591
Small refactoring
mohdsayed May 28, 2024
51938c0
Assign target in the chrome.debugger event.
amovar18 May 28, 2024
fa3804d
Add comments.
amovar18 May 29, 2024
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Make multitab work
  • Loading branch information
amovar18 committed Mar 29, 2024
commit 17b162ef8b809bece803f6aa2f3586117256fda2
10 changes: 10 additions & 0 deletions packages/extension/src/contentScript/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,11 @@ class WebpageContentScript {
*/
tabId: number | null = null;

/**
* Main frame id.
*/
frameId: string | null = null;

/**
* TabId of the current Tab
*/
Expand Down Expand Up @@ -141,6 +146,7 @@ class WebpageContentScript {

if (message?.payload?.type === 'SERVICEWORKER::WEBPAGE::TABID_STORAGE') {
this.tabId = message.payload.tabId;
this.frameId = message.payload.frameId;
}

if (message?.payload?.type === 'DEVTOOL::WEBPAGE::GET_JS_COOKIES') {
Expand All @@ -164,11 +170,15 @@ class WebpageContentScript {
* @param tabId The tabID whose cookies have to be fetched.
*/
async getAndProcessJSCookies(tabId: string) {
if (!this.frameId) {
return;
}
//@ts-ignore
const jsCookies = await cookieStore.getAll();
await processAndStoreDocumentCookies({
tabUrl: window.location.href,
tabId,
frameId: this.frameId,
documentCookies: jsCookies,
});
}
Expand Down
97 changes: 72 additions & 25 deletions packages/extension/src/serviceWorker/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ import getQueryParams from '../utils/getQueryParams';
import reloadCurrentTab from '../utils/reloadCurrentTab';
import createCookieFromAuditsIssue from '../utils/createCookieFromAuditsIssue';
import attachCDP from './attachCDP';
import { getCurrentTabId } from '../utils/getCurrentTabId';

let cookieDB: CookieDatabase | null = null;
let syncCookieStore: SynchnorousCookieStore | undefined;
Expand Down Expand Up @@ -79,6 +78,7 @@ const ALLOWED_EVENTS = [
'Audits.issueAdded',
'Network.requestWillBeSent',
'Page.frameAttached',
'Page.frameNavigated',
];

const tabIdToFrames: {
Expand All @@ -98,12 +98,13 @@ chrome.tabs.onCreated.addListener(async (tab) => {
unParsedResponseHeaders[tab.id.toString()] = {};
requestIdToCDPURLMapping[tab.id.toString()] = {};
auditsIssueForTab[tab.id.toString()] = {};
tabIdToFrames[tab.id.toString()] = new Set();

if (!syncCookieStore) {
syncCookieStore = new SynchnorousCookieStore();
}

const targets = await chrome.debugger.getTargets();

if (tabMode && tabMode !== 'unlimited') {
const doesTabExist = tabToRead;
if (
Expand All @@ -116,14 +117,18 @@ chrome.tabs.onCreated.addListener(async (tab) => {
tabToRead = tab.id.toString();
syncCookieStore?.addTabData(tab.id);

const targets = await chrome.debugger.getTargets();
const currentTab = targets.filter(
({ tabId }) => tabId && tab.id && tabId === tab.id
);

tabIdToFrames[tab.id.toString()].add(currentTab[0].id);
syncCookieStore.updateFrameIdSet(tab.id, currentTab[0].id);
} else {
syncCookieStore?.addTabData(tab.id);
const currentTab = targets.filter(
({ tabId }) => tabId && tab.id && tabId === tab.id
);

syncCookieStore.updateFrameIdSet(tab.id, currentTab[0].id);
}
});

Expand All @@ -132,6 +137,12 @@ chrome.tabs.onCreated.addListener(async (tab) => {
* @see https://developer.chrome.com/docs/extensions/reference/api/tabs#event-onRemoved
*/
chrome.tabs.onRemoved.addListener((tabId) => {
delete unParsedRequestHeaders[tabId];
delete unParsedResponseHeaders[tabId];
delete requestIdToCDPURLMapping[tabId];
delete auditsIssueForTab[tabId];
delete tabIdToFrames[tabId];

syncCookieStore?.removeTabData(tabId);
});

Expand Down Expand Up @@ -198,6 +209,13 @@ chrome.tabs.onUpdated.addListener(async (tabId, changeInfo, tab) => {

if (changeInfo.status === 'loading' && tab.url) {
syncCookieStore?.removeCookieData(tabId);

syncCookieStore?.updateFrameIdSet(
tabId,
(await chrome.debugger.getTargets()).filter(
(target) => target.tabId && target.tabId === tabId
)[0].id
);
}

try {
Expand All @@ -206,6 +224,9 @@ chrome.tabs.onUpdated.addListener(async (tabId, changeInfo, tab) => {
payload: {
type: 'SERVICEWORKER::WEBPAGE::TABID_STORAGE',
tabId,
frameId: (
await chrome.debugger.getTargets()
).filter((target) => target.tabId && target.tabId === tabId)[0].id,
},
});
} catch (error) {
Expand All @@ -222,15 +243,6 @@ chrome.tabs.onUpdated.addListener(async (tabId, changeInfo, tab) => {
} catch (error) {
//Fail silently
}
if (!tab.url) {
return;
}

syncCookieStore?.updateUrl(tabId, tab.url);

if (changeInfo.status === 'loading' && tab.url) {
syncCookieStore?.removeCookieData(tabId);
}
});

/**
Expand Down Expand Up @@ -331,9 +343,10 @@ chrome.debugger.onEvent.addListener(async (source, method, params) => {
return;
}

let tabId = await getCurrentTabId();
let tabId = '';
try {
const targets = await chrome.debugger.getTargets();

await Promise.all(
targets.map(async ({ id }) => {
await chrome.debugger.attach({ targetId: id }, '1.3');
Expand All @@ -348,6 +361,38 @@ chrome.debugger.onEvent.addListener(async (source, method, params) => {

if (source?.tabId) {
tabId = source?.tabId?.toString();
} else if (source.targetId && method !== 'Page.frameAttached') {
const tab = Object.keys(tabIdToFrames).filter(
(key) => source.targetId && tabIdToFrames[key].has(source.targetId)
);
tabId = tab[0];
}

if (method === 'Page.frameAttached' && params) {
const frameData = params as Protocol.Page.FrameAttachedEvent;
if (source.tabId) {
syncCookieStore?.updateFrameIdSet(source.tabId, frameData.frameId);
} else if (source.targetId) {
Object.values(syncCookieStore?.tabs ?? {}).forEach(({ frameIdSet }) => {
if (source.targetId && frameIdSet.has(source.targetId)) {
frameIdSet.add(frameData.frameId);
}
});
}
}

if (method === 'Page.frameNavigated' && params) {
const {
frame: { parentId = '', id },
} = params as Protocol.Page.FrameNavigatedEvent;

if (parentId) {
Object.values(syncCookieStore?.tabs ?? {}).forEach(({ frameIdSet }) => {
if (frameIdSet.has(parentId)) {
frameIdSet.add(id);
}
});
}
}

if (!tabId) {
Expand All @@ -364,17 +409,6 @@ chrome.debugger.onEvent.addListener(async (source, method, params) => {
cookieDB = await fetchDictionary();
}

if (method === 'Page.frameAttached' && params) {
const frameData = params as Protocol.Page.FrameAttachedEvent;
if (frameData?.parentFrameId) {
Object.values(tabIdToFrames).forEach((setOfFrames) => {
if (setOfFrames.has(frameData.parentFrameId)) {
setOfFrames.add(frameData.frameId);
}
});
}
}

if (method === 'Network.requestWillBeSent' && params) {
const request = params as Protocol.Network.RequestWillBeSentEvent;
// To get domain from the request URL if not given in the cookie line.
Expand Down Expand Up @@ -770,6 +804,19 @@ chrome.runtime.onMessage.addListener(async (request) => {
})
);
}

if (request?.type === 'SERVICE_WORKER_DEVTOOLS_GET_FRAME_IDS') {
const requestedTabId = request?.payload?.tabId;
chrome.runtime.sendMessage({
type: 'ServiceWorker::DevTools::NEW_FRAMES_DATA',
payload: {
tabId: requestedTabId,
frames: Array.from(
syncCookieStore?.tabs[requestedTabId].frameIdSet ?? new Set()
),
},
});
}
});

/**
Expand Down
15 changes: 5 additions & 10 deletions packages/extension/src/store/synchnorousCookieStore.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,7 @@ class SynchnorousCookieStore {
devToolsOpenState: boolean;
popupOpenState: boolean;
newUpdates: number;
frameIdSet: {
[frameUrl: string]: string[];
};
frameIdSet: Set<string>;
};
} = {};

Expand Down Expand Up @@ -190,7 +188,7 @@ class SynchnorousCookieStore {
* @param {number} tabId Tab id.
* @returns {string | null} The url of the tab if exists else null.
*/
getFrameIDSet(tabId: number): { [frameUrl: string]: string[] } | null {
getFrameIDSet(tabId: number): Set<string> | null {
if (!this.tabs[tabId]) {
return null;
}
Expand All @@ -201,16 +199,13 @@ class SynchnorousCookieStore {
/**
* Update FrameId set for a given url for a given tab.
* @param {number} tabId The url whose url needs to be update.
* @param {string} frameUrl The URL where frameId needs to be added.
* @param {string} frameIdToAdd The new frameId to be added.
*/
updateFrameIdSet(tabId: number, frameUrl: string, frameIdToAdd: string) {
updateFrameIdSet(tabId: number, frameIdToAdd: string) {
if (!this.tabs[tabId]) {
return;
} else {
this.tabs[tabId].frameIdSet[frameUrl] = [
...new Set([...this.tabs[tabId].frameIdSet[frameUrl], frameIdToAdd]),
];
this.tabs[tabId].frameIdSet.add(frameIdToAdd);
}
}

Expand Down Expand Up @@ -336,7 +331,7 @@ class SynchnorousCookieStore {
devToolsOpenState: false,
popupOpenState: false,
newUpdates: 0,
frameIdSet: {},
frameIdSet: new Set(),
};
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,12 +32,14 @@ import { fetchDictionary } from './fetchCookieDictionary';
interface ProcessAndStoreDucmentCookies {
tabUrl: string;
tabId: string;
frameId: string;
documentCookies: ParsedCookie[];
}

const processAndStoreDocumentCookies = async ({
tabUrl,
tabId,
frameId,
documentCookies,
}: ProcessAndStoreDucmentCookies) => {
try {
Expand Down Expand Up @@ -80,7 +82,7 @@ const processAndStoreDocumentCookies = async ({
url: tabUrl,
headerType: 'javascript', // @todo Update headerType name.
isFirstParty: isFirstPartyCookie || null,
frameIdList: [0],
frameIdList: [frameId],
};
}
);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,66 @@ export const Provider = ({ children }: PropsWithChildren) => {
[]
);

useEffect(() => {
if (!tabFrames) {
return;
}

chrome.runtime.sendMessage({
type: 'SERVICE_WORKER_DEVTOOLS_GET_FRAME_IDS',
payload: {
tabId: chrome.devtools.inspectedWindow.tabId,
},
});
}, [tabFrames]);

/**
* Add more tabframes to the existing ones.
*/
const addMoreTabIds = useCallback(
async (tabIds: string[]) => {
if (!tabFrames) {
return;
}

const modifiedTabFrames = tabFrames;

await Promise.all(
tabIds.map(async (id) => {
//@ts-ignore
try {
const frames = await chrome.debugger.sendCommand(
{ targetId: id },
'Page.getFrameTree'
);
if (
//@ts-ignore
frames?.frameTree?.frame?.securityOrigin &&
//@ts-ignore
frames.frameTree.frame.id
) {
modifiedTabFrames[
//@ts-ignore
frames?.frameTree?.frame?.securityOrigin
].frameIds = [
...new Set([
//@ts-ignore
...modifiedTabFrames[frames.frameTree.frame.securityOrigin]
.frameIds,
//@ts-ignore
frames.frameTree.frame.id,
]),
];
}
} catch (error) {
// fail silently
}
})
);
setTabFrames(modifiedTabFrames);
},
[tabFrames]
);
/**
* Stores object with frame URLs as keys and boolean values indicating if the frame contains cookies.
*/
Expand Down Expand Up @@ -329,6 +389,7 @@ export const Provider = ({ children }: PropsWithChildren) => {
cookieData?: TabCookies;
tabToRead?: string;
tabMode?: string;
frames?: string[];
};
}) => {
if (message.type === 'ServiceWorker::SET_TAB_TO_READ') {
Expand Down Expand Up @@ -378,6 +439,10 @@ export const Provider = ({ children }: PropsWithChildren) => {
}
}

if (message.type === 'ServiceWorker::DevTools::NEW_FRAMES_DATA') {
await addMoreTabIds(message.payload.frames ?? []);
}

if (message.type === 'ServiceWorker::DevTools::TABS_RELOADED') {
setSettingsChanged(false);
}
Expand All @@ -388,7 +453,7 @@ export const Provider = ({ children }: PropsWithChildren) => {
return () => {
chrome.runtime.onMessage.removeListener(listener);
};
}, [getAllFramesForCurrentTab, tabId, setSettingsChanged]);
}, [getAllFramesForCurrentTab, tabId, setSettingsChanged, addMoreTabIds]);

const tabUpdateListener = useCallback(
async (_tabId: number, changeInfo: chrome.tabs.TabChangeInfo) => {
Expand Down