Early version of a proper file-driven pipeline with 2 example queries

README.md

@@ -1,2 +1,7 @@
-# rum-insights-data
-Queries and data for RUM insights
+# RUM Insights
+
+Queries and data from the RUM Archive for use in RUM Insights visualizations.
+
+To get started, look at `/docs` and look at `pipeline.js` (note: this needs manual steps to setup proper BigQuery API credentials that are currently undocumented).
+
+For now, the idea is to use the data in `/data-output` directly as consumable "API endpoints" for use in RUM Insights visualizations. 

data-output/LCP_visibilitystate_devicetype.json

@@ -0,0 +1 @@
+[{"client":"unknown","date":"2023_07_01","visibilitystate":"hidden","timestamp":"1688169600000","histogram":[[198,1],[267,6],[363,13],[440,12],[559,8],[658,4],[765,9],[836,6],[923,5],[1041,3],[1129,4],[1258,4],[1351,4],[1453,7],[1553,6],[1646,3],[1777,6],[1833,8],[1913,4],[2050,6],[2139,5],[2260,4],[2353,4],[2442,4],[2557,4],[2624,2],[2738,5],[2871,5],[2940,2],[3014,1],[3132,3],[3247,4],[3332,5],[3436,4],[3547,2],[3738,2],[3958,4],[4066,4],[4147,1],[4360,2],[4663,2],[4739,5],[4956,3],[5036,2],[5186,1],[5246,5],[5382,5],[5462,4],[5547,6],[5635,3],[5745,1],[5862,3],[5950,5],[6000,1],[6136,3],[6226,3],[6347,2],[6455,2],[6546,2],[6649,2],[6830,1],[6941,2],[7039,1],[7161,2],[7231,3],[7364,2],[7518,1],[7727,1],[7883,1],[7980,1],[8092,1],[8148,2],[8582,1],[8662,3],[8786,1],[8896,1],[8966,1],[9061,1],[9467,1],[9542,1],[9666,1],[9954,1]]},{"client":"unknown","date":"2023_07_01","visibilitystate":"partial","timestamp":"1688169600000","histogram":[[2198,1],[2654,1],[3625,1],[4534,1],[5225,1]]},{"client":"unknown","date":"2023_07_01","visibilitystate":"visible","timestamp":"1688169600000","histogram":[[178,11],[273,777],[357,2638],[446,3477],[546,2930],[648,2517],[750,2385],[851,2576],[954,4152],[1055,5903],[1152,6255],[1250,6480],[1350,5862],[1448,4905],[1548,4193],[1649,3591],[1751,3290],[1849,2945],[1950,2630],[2050,2612],[2149,2054],[2247,1885],[2348,1554],[2450,1439],[2550,1280],[2651,1197],[2752,1139],[2855,1043],[2949,970],[3052,991],[3150,931],[3253,900],[3348,921],[3451,881],[3552,814],[3650,793],[3752,720],[3848,678],[3953,667],[4051,598],[4155,592],[4252,528],[4350,482],[4449,434],[4551,446],[4653,400],[4752,399],[4847,373],[4949,347],[5051,347],[5148,267],[5249,241],[5349,287],[5456,229],[5548,232],[5650,226],[5754,217],[5853,194],[5960,199],[6054,171],[6145,176],[6254,165],[6349,166],[6448,157],[6547,157],[6643,158],[6748,149],[6849,134],[6951,123],[7048,119],[7146,111],[7247,128],[7354,118],[7457,106],[7549,92],[7647,96],[7749,100],[7844,89],[7954,103],[8052,89],[8150,87],[8257,85],[8343,70],[8453,76],[8547,76],[8655,71],[8755,73],[8849,56],[8946,54],[9053,53],[9146,51],[9259,53],[9363,47],[9449,45],[9550,30],[9652,44],[9742,45],[9847,26],[9952,34]]},{"client":"desktop","date":"2023_07_01","visibilitystate":"hidden","timestamp":"1688169600000","histogram":[[81,352],[157,2702],[254,5831],[353,9645],[448,13187],[551,15536],[649,16727],[750,17467],[849,17607],[951,17161],[1047,16654],[1147,16241],[1247,15155],[1351,14282],[1449,13327],[1549,12805],[1650,11815],[1751,11153],[1852,10301],[1946,9625],[2052,9261],[2151,8392],[2246,7895],[2349,7358],[2446,6798],[2545,6517],[2648,5853],[2751,5536],[2847,5268],[2948,4774],[3050,4540],[3150,4211],[3247,4007],[3345,3666],[3448,3425],[3547,3254],[3650,3178],[3746,2900],[3847,2770],[3948,2577],[4050,2459],[4148,2300],[4251,2211],[4346,2091],[4447,2039],[4551,1835],[4650,1787],[4747,1658],[4850,1638],[4956,1507],[5050,1421],[5148,1358],[5250,1309],[5348,1309],[5448,1236],[5549,1172],[5646,1107],[5750,1004],[5851,980],[5951,880],[6052,909],[6149,871],[6246,899],[6352,875],[6451,849],[6547,737],[6648,755],[6749,671],[6850,677],[6948,660],[7049,637],[7145,585],[7250,572],[7349,546],[7451,505],[7551,530],[7651,468],[7750,495],[7849,482],[7952,499],[8048,432],[8148,457],[8249,453],[8351,438],[8449,406],[8552,389],[8657,377],[8748,355],[8849,380],[8953,330],[9048,300],[9147,313],[9247,321],[9350,316],[9450,281],[9548,234],[9650,282],[9753,297],[9853,270],[9957,238]]},{"client":"desktop","date":"2023_07_01","visibilitystate":"partial","timestamp":"1688169600000","histogram":[[79,10],[162,163],[256,339],[355,567],[451,721],[552,1005],[655,1290],[749,1300],[848,1303],[955,1393],[1050,1423],[1150,1494],[1254,1497],[1349,1542],[1445,1580],[1546,1508],[1651,1584],[1751,1501],[1852,1415],[1947,1394],[2048,1391],[2152,1316],[2253,1261],[2347,1253],[2450,1265],[2555,1160],[2650,1155],[2747,1056],[2850,1112],[2953,978],[3052,1000],[3142,988],[3246,958],[3349,929],[3450,944],[3553,900],[3648,905],[3751,856],[3851,811],[3948,835],[4050,797],[4150,847],[4246,728],[4353,756],[4445,737],[4551,703],[4650,703],[4752,618],[4847,664],[4949,615],[5051,619],[5153,647],[5251,585],[5347,550],[5446,598],[5553,563],[5652,539],[5750,490],[5847,475],[5953,534],[6050,473],[6150,459],[6244,478],[6348,440],[6446,468],[6551,418],[6649,403],[6751,412],[6850,349],[6948,390],[7049,372],[7151,369],[7246,385],[7351,378],[7455,340],[7552,359],[7646,324],[7749,334],[7844,308],[7952,328],[8050,322],[8147,284],[8252,312],[8351,297],[8450,307],[8555,302],[8654,260],[8747,290],[8848,282],[8955,282],[9051,276],[9152,235],[9248,230],[9351,255],[9448,213],[9553,228],[9648,230],[9753,242],[9850,216],[9949,202]]},{"client":"desktop","date":"2023_07_01","visibilitystate":"visible","timestamp":"1688169600000","histogram":[[72,35716],[157,213354],[257,372205],[353,558096],[452,685782],[551,765026],[652,809950],[751,835969],[851,845104],[952,836582],[1050,818116],[1149,794817],[1249,765832],[1350,737831],[1449,708536],[1548,676458],[1649,641392],[1749,601048],[1848,560963],[1950,520678],[2048,486698],[2149,448075],[2249,413213],[2349,381620],[2446,354543],[2547,327915],[2649,304354],[2747,283634],[2849,264227],[2949,246995],[3047,230420],[3152,216536],[3249,202756],[3350,192137],[3449,180006],[3548,169458],[3650,159987],[3749,151924],[3847,143239],[3952,135917],[4055,138491],[4145,123912],[4249,114184],[4351,107438],[4451,101117],[4549,95814],[4649,91210],[4749,85197],[4847,81037],[4950,77282],[5050,72404],[5149,68906],[5249,65395],[5349,62294],[5449,59726],[5551,56525],[5648,54042],[5749,51678],[5852,48938],[5948,47403],[6048,44737],[6150,43446],[6249,41451],[6351,39493],[6452,38168],[6551,36632],[6652,34794],[6753,33451],[6852,32269],[6950,30251],[7047,29290],[7152,28137],[7254,27277],[7355,25824],[7450,24934],[7551,24565],[7651,23446],[7749,22626],[7852,21943],[7950,21134],[8048,20834],[8149,20154],[8247,19148],[8349,18661],[8449,18355],[8550,17803],[8652,16993],[8753,17075],[8846,16252],[8948,15908],[9049,15675],[9153,14785],[9251,14229],[9350,14093],[9451,13495],[9550,12841],[9651,12430],[9750,12171],[9848,11521],[9953,11310]]},{"client":"mobile","date":"2023_07_01","visibilitystate":"hidden","timestamp":"1688169600000","histogram":[[75,78],[169,1906],[253,7392],[356,12378],[452,16677],[552,20570],[652,22428],[749,23767],[846,23774],[949,23818],[1049,23679],[1149,22997],[1252,22144],[1351,21209],[1450,20215],[1548,19167],[1647,18041],[1749,17190],[1849,16254],[1946,15237],[2047,14396],[2147,13367],[2251,12463],[2349,11728],[2451,10998],[2551,10347],[2652,9730],[2745,9068],[2848,8419],[2946,8047],[3049,7504],[3151,7195],[3249,6785],[3351,6353],[3449,5903],[3550,5612],[3650,5329],[3747,5087],[3849,4766],[3949,4509],[4050,4329],[4151,4132],[4252,3879],[4350,3590],[4450,3491],[4551,3329],[4647,3112],[4750,3029],[4849,2864],[4948,2762],[5053,2617],[5151,2391],[5255,2438],[5351,2262],[5449,2166],[5545,2070],[5654,2065],[5749,1877],[5848,1774],[5954,1701],[6048,1659],[6149,1566],[6251,1515],[6347,1378],[6452,1461],[6554,1298],[6653,1311],[6750,1236],[6853,1182],[6951,1135],[7049,1069],[7152,1093],[7250,1051],[7350,975],[7448,927],[7548,886],[7646,850],[7753,861],[7851,856],[7950,787],[8046,747],[8150,719],[8254,685],[8355,710],[8447,654],[8549,648],[8646,653],[8748,611],[8847,573],[8951,652],[9051,563],[9153,586],[9249,496],[9346,515],[9450,481],[9549,481],[9650,425],[9749,434],[9854,423],[9944,405]]},{"client":"mobile","date":"2023_07_01","visibilitystate":"partial","timestamp":"1688169600000","histogram":[[84,4],[175,68],[258,307],[353,541],[451,726],[555,954],[654,1181],[750,1234],[851,1450],[950,1490],[1053,1508],[1149,1499],[1251,1509],[1349,1490],[1450,1423],[1548,1451],[1651,1405],[1750,1355],[1848,1263],[1949,1239],[2046,1205],[2151,1175],[2248,1065],[2348,1053],[2451,1030],[2542,972],[2650,891],[2748,943],[2848,883],[2953,854],[3050,785],[3155,775],[3251,719],[3347,676],[3448,675],[3556,675],[3653,640],[3751,629],[3845,567],[3947,561],[4048,526],[4149,532],[4247,526],[4352,521],[4448,489],[4544,468],[4652,480],[4753,436],[4852,460],[4953,409],[5051,412],[5150,412],[5254,390],[5348,372],[5448,383],[5549,378],[5650,337],[5751,333],[5844,321],[5948,313],[6052,333],[6150,354],[6250,288],[6354,283],[6450,298],[6548,270],[6653,261],[6753,279],[6851,266],[6947,254],[7048,263],[7148,238],[7250,219],[7354,220],[7445,227],[7549,212],[7647,214],[7749,195],[7848,223],[7954,210],[8050,183],[8154,185],[8250,202],[8351,169],[8451,167],[8548,187],[8650,179],[8751,166],[8848,166],[8947,160],[9049,160],[9151,157],[9248,146],[9347,156],[9453,142],[9550,142],[9650,153],[9746,133],[9856,121],[9949,163]]},{"client":"mobile","date":"2023_07_01","visibilitystate":"visible","timestamp":"1688169600000","histogram":[[58,3183],[165,55035],[257,202975],[359,390065],[452,540301],[551,623606],[650,684417],[749,718160],[849,729338],[949,725035],[1049,708171],[1150,692117],[1250,670550],[1350,645005],[1450,613473],[1551,582636],[1648,552593],[1748,525761],[1850,499227],[1947,475828],[2048,452549],[2147,430204],[2248,407311],[2349,386960],[2449,366391],[2548,347277],[2650,329021],[2749,312167],[2848,293449],[2949,279461],[3050,264495],[3152,249802],[3249,238279],[3347,226555],[3446,215641],[3547,206358],[3647,196224],[3747,187509],[3849,178371],[3950,169428],[4049,161330],[4149,154184],[4249,146342],[4349,139869],[4450,132740],[4548,126511],[4650,120140],[4749,114766],[4852,109540],[4946,104780],[5048,99810],[5151,94875],[5252,90423],[5351,86971],[5449,82678],[5554,78966],[5651,76539],[5747,72566],[5849,69843],[5950,66895],[6050,64494],[6152,62529],[6248,59894],[6349,56990],[6450,55171],[6546,52821],[6651,50634],[6750,48656],[6848,47131],[6946,45126],[7052,42815],[7153,41525],[7252,39861],[7351,38509],[7450,36654],[7548,35538],[7649,34145],[7746,32618],[7848,31568],[7950,30627],[8047,29527],[8149,28470],[8251,28125],[8348,26802],[8450,25733],[8549,24982],[8653,23937],[8751,22960],[8855,22494],[8951,21732],[9049,20924],[9146,20388],[9250,19607],[9350,18738],[9444,18569],[9552,17826],[9648,17364],[9750,16976],[9846,16290],[9951,15686]]},{"client":"tablet","date":"2023_07_01","visibilitystate":"hidden","timestamp":"1688169600000","histogram":[[98,1],[175,12],[250,55],[358,112],[454,206],[549,290],[653,438],[750,540],[853,586],[953,647],[1051,715],[1151,802],[1251,857],[1351,939],[1453,914],[1553,978],[1651,1068],[1750,987],[1849,1036],[1948,935],[2048,932],[2147,992],[2248,888],[2351,866],[2449,841],[2547,755],[2648,714],[2754,682],[2845,615],[2947,645],[3041,550],[3149,541],[3248,512],[3354,513],[3450,486],[3550,453],[3648,434],[3746,421],[3846,415],[3947,404],[4045,414],[4158,345],[4252,326],[4354,347],[4455,339],[4547,331],[4647,310],[4751,332],[4854,240],[4949,260],[5047,239],[5149,244],[5245,228],[5348,223],[5448,192],[5550,214],[5646,194],[5744,154],[5847,175],[5951,175],[6050,165],[6152,148],[6254,133],[6349,129],[6453,138],[6541,114],[6647,123],[6748,112],[6848,99],[6945,110],[7048,111],[7149,87],[7246,98],[7354,111],[7451,87],[7546,71],[7654,86],[7753,96],[7857,77],[7949,96],[8046,70],[8145,76],[8258,70],[8353,49],[8446,63],[8558,61],[8657,55],[8749,61],[8847,60],[8951,44],[9046,53],[9152,41],[9251,40],[9347,40],[9448,44],[9543,46],[9651,37],[9737,40],[9856,28],[9947,46]]},{"client":"tablet","date":"2023_07_01","visibilitystate":"partial","timestamp":"1688169600000","histogram":[[157,1],[295,1],[445,3],[544,6],[667,17],[733,15],[861,17],[948,18],[1049,20],[1157,20],[1253,24],[1348,28],[1445,33],[1551,30],[1646,36],[1742,21],[1851,31],[1954,26],[2064,12],[2152,26],[2251,32],[2349,22],[2452,25],[2553,27],[2656,20],[2756,14],[2855,20],[2947,24],[3054,22],[3154,19],[3235,19],[3345,23],[3451,19],[3552,15],[3641,17],[3754,14],[3858,10],[3951,18],[4051,15],[4145,21],[4259,13],[4349,14],[4446,11],[4555,9],[4649,13],[4762,17],[4856,15],[4938,14],[5045,19],[5155,15],[5256,12],[5343,15],[5441,14],[5539,10],[5647,15],[5749,10],[5846,8],[5961,10],[6052,8],[6147,8],[6257,14],[6343,9],[6441,11],[6558,7],[6661,7],[6729,5],[6841,10],[6941,7],[7055,4],[7134,11],[7245,8],[7358,7],[7439,7],[7559,11],[7641,8],[7739,11],[7835,7],[7958,4],[8052,11],[8114,4],[8265,4],[8351,4],[8459,7],[8556,6],[8654,5],[8751,5],[8857,6],[8948,5],[9070,6],[9154,5],[9253,11],[9331,6],[9468,8],[9565,5],[9670,2],[9754,9],[9853,3],[9940,3]]},{"client":"tablet","date":"2023_07_01","visibilitystate":"visible","timestamp":"1688169600000","histogram":[[88,21],[165,612],[261,2585],[361,5781],[454,9585],[554,13426],[654,17165],[750,19672],[848,21033],[948,21351],[1049,21900],[1148,22150],[1247,22139],[1347,21909],[1447,21556],[1548,20942],[1650,20103],[1747,19287],[1847,18374],[1948,17863],[2049,16882],[2149,16303],[2246,15391],[2351,14916],[2447,14061],[2546,13269],[2651,12596],[2750,12176],[2849,11323],[2951,10862],[3049,10256],[3151,9841],[3250,9653],[3351,8809],[3450,8647],[3551,8496],[3649,8181],[3751,7733],[3850,7507],[3952,7288],[4053,6942],[4151,6774],[4251,6462],[4352,6290],[4445,6029],[4549,5864],[4651,5587],[4751,5390],[4849,5063],[4949,4795],[5048,4757],[5149,4502],[5252,4260],[5348,4117],[5454,4061],[5550,3901],[5649,3755],[5750,3573],[5850,3477],[5951,3377],[6051,3334],[6149,3129],[6248,3098],[6350,3150],[6451,2895],[6549,2811],[6648,2761],[6751,2706],[6847,2608],[6950,2494],[7050,2438],[7150,2414],[7255,2352],[7349,2197],[7453,2262],[7552,2020],[7649,2026],[7750,1925],[7852,1887],[7949,1901],[8047,1787],[8151,1743],[8248,1661],[8348,1633],[8448,1619],[8546,1551],[8649,1527],[8750,1445],[8852,1426],[8946,1379],[9052,1382],[9154,1362],[9246,1258],[9350,1165],[9447,1194],[9548,1192],[9654,1146],[9750,1114],[9849,1135],[9957,1063]]}]

docs/README.md

@@ -0,0 +1,34 @@
+# RUM Insights data pipeline
+
+The raw RUM Archive data is published in Google BigQuery.
+To generate the RUM Insights visualizations, we don't want to constantly query that live for every user though.
+
+This repository contains queries and scripts to create a "cached" version of the needed data that is stored separately (currently just on github) and can be accessed by the visualizations as a ready-to-use JSON file. 
+
+This process contains several steps which are detailed below:
+1. Figure out which queries to run.
+2. Run the queries through BigQuery to get the full raw results. Store these in a cache.
+3. Process the raw results into a more easy-to-use output format for the RUM Insights visualizations. 
+
+
+## 1. Determine queries
+
+We have several queries to run, typically 1 per visualization (though some can share query outputs). 
+Some queries need the full temporal range since the RUM Archive start, while others only need the latest data (last available month). 
+
+The queries are stored as JSONL in `/queries` and contain a combination of the raw SQL commands (together with some placeholder text of the form `{{TO_BE_REPLACED}}` for certain variables (typically date ranges)) and some query metadata (e.g., serializable (file)name, type of query, description, enabled/disabled, etc.).
+
+```
+Note: the we use a non-standard JSON variant `.jsonl` that allows newlines in strings to make the SQL queries more readable. We work around this by removing newlines from the read files before passing them to the JSON parser. 
+```
+
+TODO: discuss looking at cached previous results and only run queries for newer data
+
+## 2. Run BigQuery
+
+TODO: note that this repo does not contain full instructions on how to setup the bigquery API credentials etc. Currently this is being done periodically and manually on a local PC.
+
+## 3. Transform results
+
+TODO: discuss that one goal is to get output format similar to HTTPArchive so we can more easily compare data across the datasets in the same visualizations
+