Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aria2c, but for real now #220

Draft
wants to merge 69 commits into
base: v3.0
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
0cbc962
first draft of DownloadManager
lukaarma Sep 5, 2020
5334293
first draft of Decrypter
lukaarma Sep 5, 2020
67573fc
installed WebSocket and updated Tmp
lukaarma Sep 5, 2020
685fa27
added debug and best quality flags
lukaarma Sep 5, 2020
0d8b420
changed UserAgent version
lukaarma Sep 5, 2020
6c0e37a
created WebSocket/Aria2c errors
lukaarma Sep 5, 2020
29a6fab
minor formatting changes
lukaarma Sep 5, 2020
d29bd54
very simple test for SIGINT
lukaarma Sep 5, 2020
5350bc3
added debug logging
lukaarma Sep 5, 2020
0c65ff7
added parsing of m3u8 file down to a list of links
lukaarma Sep 5, 2020
903f2bf
updated destreamer to use the new download method
lukaarma Sep 5, 2020
1dff41c
Merge branch 'master' of https://github.com/snobu/destreamer into ari…
lukaarma Sep 7, 2020
6e874f5
exclude debug launch config
lukaarma Sep 7, 2020
9e25870
updated packages
lukaarma Sep 7, 2020
d037b7c
renamed and finished decrypter
lukaarma Sep 9, 2020
8b61f86
added debug logging
lukaarma Sep 9, 2020
aa21e54
added quality option
lukaarma Sep 9, 2020
a185f51
fixed typo and added comment
lukaarma Sep 9, 2020
96f4c90
- removed useless function/properties
lukaarma Sep 9, 2020
ec099e9
- fixed progress bar not updating
lukaarma Sep 9, 2020
f1476ff
done decryption/merging of video/audio/sub traks
lukaarma Sep 9, 2020
a93b328
updating comments
lukaarma Sep 9, 2020
e9dea14
removed bug in quality selection
lukaarma Sep 9, 2020
16a8532
removed useless old code
lukaarma Sep 9, 2020
6f082e1
marked unused code
lukaarma Sep 9, 2020
796753f
change user agent
lukaarma Sep 12, 2020
6c8628e
code cleanup
lukaarma Sep 19, 2020
7cab44a
added debug statement
lukaarma Sep 19, 2020
9453458
installed portfinder
lukaarma Sep 19, 2020
3e472f9
created error for no port aviable
lukaarma Sep 19, 2020
8df5155
implemented port finding
lukaarma Sep 19, 2020
c9c9fef
removed random useless linting test
lukaarma Sep 19, 2020
af4725c
fixed linting errors
lukaarma Sep 19, 2020
38edbad
check for aria existance
lukaarma Sep 20, 2020
482a506
fixed bug on hanging on shutdown
lukaarma Sep 20, 2020
95c7150
improved shutdown sequence
lukaarma Sep 20, 2020
14cfe7c
typo fix
lukaarma Sep 24, 2020
c7e0415
process.exit on uncaught exceptions
lukaarma Sep 24, 2020
502565d
better webSocket initialization
lukaarma Sep 24, 2020
020518e
minor comments and variables name changes
lukaarma Sep 25, 2020
331efd9
fix for *nix platforms
lukaarma Sep 25, 2020
b89e041
workaround for listeners not removed
lukaarma Sep 27, 2020
8848f29
Update README.md
rohit404404 Oct 3, 2020
389be33
Update README.md
rohit404404 Oct 3, 2020
ec24ff9
Merge pull request #242 from rohit404404/patch-1
lukaarma Oct 3, 2020
22968f4
Misc fixes (#244)
lukaarma Oct 8, 2020
15c4203
Update report-trouble.md (#240)
lukaarma Oct 8, 2020
2a3c3eb
fixed buffer maxLenght exceded on aria2c daemon
lukaarma Oct 9, 2020
3249759
Merge branch 'master' of https://github.com/snobu/destreamer into ari…
lukaarma Oct 11, 2020
85f3bea
added access token refresh logic (closes #255)
lukaarma Oct 11, 2020
41206a9
fixed stupid session refreshing typo
lukaarma Oct 11, 2020
1763dc8
changed exec in favour of spawn for aria2c
lukaarma Oct 11, 2020
a179bdc
Update Errors.ts
rpaulucci3 Oct 17, 2020
eacb2b6
Merge pull request #265 from rpaulucci3/patch-1
lukaarma Oct 17, 2020
4a45c68
Fix group download, added 100 video limit
snobu Nov 14, 2020
0f2d902
Fix verbose logging when we get JSON back
snobu Nov 14, 2020
461df7b
Fix for group download IRL this time
snobu Nov 14, 2020
0b4c900
Update README.md
snobu Dec 2, 2020
ddafc50
fixed parsing for group with more than 100 videos
lukaarma Dec 3, 2020
08364ed
new version of error retry, couldn't test it
lukaarma Dec 3, 2020
6d28b3c
updated all packages
lukaarma Dec 3, 2020
c1a0994
add aria2 to prereqs
fulminemizzega Dec 7, 2020
66a7f60
fix quotes in input file example in README.md
fulminemizzega Dec 7, 2020
195d0dd
Added `aria2` prerequisite
pastacolsugo Dec 26, 2020
1e8db6f
Merge pull request #303 from pastacolsugo/patch-1
lukaarma Jan 8, 2021
6d8f3c6
Fix return HTTP 403 reason with or without verbose (#316)
snobu Jan 26, 2021
3fe64d1
Merge branch 'aria2c_forRealNow' into patch-1
lukaarma Feb 9, 2021
cea18bb
Merge pull request #289 from fulminemizzega/patch-1
lukaarma Feb 9, 2021
ffd76ba
Fix a small typo in terminal output (#363)
MahmoudAFarag Apr 7, 2021
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Misc fixes (#244)
* cleared up docs

* fix 401 bug for images thumbnail
  • Loading branch information
lukaarma committed Oct 8, 2020
commit 22968f42ea4970f20962fed5ab9bb355f0f70a0a
28 changes: 16 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,31 +108,35 @@ Options:
--help Show help [boolean]
--version Show version number [boolean]
--username, -u The username used to log into Microsoft Stream (enabling this will fill in the email field for
you) [string]
--videoUrls, -i List of video urls [array]
you). [string]
--videoUrls, -i List of urls to videos or Microsoft Stream groups. [array]
--inputFile, -f Path to text file containing URLs and optionally outDirs. See the README for more on outDirs.
[string]
--outputDirectory, -o The directory where destreamer will save your downloads. [string] [default: "videos"]
--outputTemplate, -t The template for the title. See the README for more info.
[string] [default: "{title} - {publishDate} {uniqueId}"]
--outputDirectory, -o The directory where destreamer will save your downloads [string] [default: "videos"]
--keepLoginCookies, -k Let Chromium cache identity provider cookies so you can use "Remember me" during login
--keepLoginCookies, -k Let Chromium cache identity provider cookies so you can use "Remember me" during login.
Must be used every subsequent time you launch Destreamer if you want to log in automatically.
[boolean] [default: false]
--noExperiments, -x Do not attempt to render video thumbnails in the console [boolean] [default: false]
--simulate, -s Disable video download and print metadata information to the console[boolean] [default: false]
--verbose, -v Print additional information to the console (use this before opening an issue on GitHub)
--noExperiments, -x Do not attempt to render video thumbnails in the console. [boolean] [default: false]
--simulate, -s Disable video download and print metadata information to the console.
[boolean] [default: false]
--closedCaptions, --cc Check if closed captions are aviable and let the user choose which one to download (will not
ask if only one aviable) [boolean] [default: false]
--noCleanup, --nc Do not delete the downloaded video file when an FFmpeg error occurs [boolean] [default: false]
--verbose, -v Print additional information to the console (use this before opening an issue on GitHub).
[boolean] [default: false]
--closedCaptions, --cc Check if closed captions are available and let the user choose which one to download (will not
ask if only one available). [boolean] [default: false]
--noCleanup, --nc Do not delete the downloaded video file when an FFmpeg error occurs.[boolean] [default: false]
--vcodec Re-encode video track. Specify FFmpeg codec (e.g. libx265) or set to "none" to disable video.
[string] [default: "copy"]
--acodec Re-encode audio track. Specify FFmpeg codec (e.g. libopus) or set to "none" to disable audio.
[string] [default: "copy"]
--format Output container format (mkv, mp4, mov, anything that FFmpeg supports)
--format Output container format (mkv, mp4, mov, anything that FFmpeg supports).
[string] [default: "mkv"]
--skip Skip download if file already exists [boolean] [default: false]
--skip Skip download if file already exists. [boolean] [default: false]
```

- both --videoUrls and --inputFile also accept Microsoft Teams Groups url so if your Organization placed the videos you are interested in a group you can copy the link and Destreamer will download all the videos it can inside it! A group url looks like this https://web.microsoftstream.com/group/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

- Passing `--username` is optional. It's there to make logging in faster (the username field will be populated automatically on the login form).

- You can use an absolute path for `-o` (output directory), for example `/mnt/videos`.
Expand Down
15 changes: 15 additions & 0 deletions src/ApiClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,11 @@ export class ApiClient {
});
}

/**
* Used to initialize/retrive the active ApiClient
*
* @param session used if initializing
*/
public static getInstance(session?: Session): ApiClient {
if (!ApiClient.instance) {
ApiClient.instance = new ApiClient(session);
Expand All @@ -51,6 +56,16 @@ export class ApiClient {
return ApiClient.instance;
}

public setSession(session: Session): void {
if (!ApiClient.instance) {
logger.warn("Trying to update ApiCient session when it's not initialized!");
}

this.session = session;

return;
}

/**
* Call Microsoft Stream API. Base URL is sourced from
* the session object and prepended automatically.
Expand Down
23 changes: 12 additions & 11 deletions src/CommandLineParser.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,12 @@ export const argv: any = yargs.options({
username: {
alias: 'u',
type: 'string',
describe: 'The username used to log into Microsoft Stream (enabling this will fill in the email field for you)',
describe: 'The username used to log into Microsoft Stream (enabling this will fill in the email field for you).',
demandOption: false
},
videoUrls: {
alias: 'i',
describe: 'List of video urls',
describe: 'List of urls to videos or Microsoft Stream groups.',
type: 'array',
demandOption: false
},
Expand All @@ -30,7 +30,7 @@ export const argv: any = yargs.options({
},
outputDirectory: {
alias: 'o',
describe: 'The directory where destreamer will save your downloads',
describe: 'The directory where destreamer will save your downloads.',
type: 'string',
default: 'videos',
demandOption: false
Expand All @@ -44,42 +44,43 @@ export const argv: any = yargs.options({
},
keepLoginCookies: {
alias: 'k',
describe: 'Let Chromium cache identity provider cookies so you can use "Remember me" during login',
describe: 'Let Chromium cache identity provider cookies so you can use "Remember me" during login.\n' +
'Must be used every subsequent time you launch Destreamer if you want to log in automatically.',
type: 'boolean',
default: false,
demandOption: false
},
noExperiments: {
alias: 'x',
describe: 'Do not attempt to render video thumbnails in the console',
describe: 'Do not attempt to render video thumbnails in the console.',
type: 'boolean',
default: false,
demandOption: false
},
simulate: {
alias: 's',
describe: 'Disable video download and print metadata information to the console',
describe: 'Disable video download and print metadata information to the console.',
type: 'boolean',
default: false,
demandOption: false
},
verbose: {
alias: 'v',
describe: 'Print additional information to the console (use this before opening an issue on GitHub)',
describe: 'Print additional information to the console (use this before opening an issue on GitHub).',
type: 'boolean',
default: false,
demandOption: false
},
closedCaptions: {
alias: 'cc',
describe: 'Check if closed captions are available and let the user choose which one to download (will not ask if only one available)',
describe: 'Check if closed captions are available and let the user choose which one to download (will not ask if only one available).',
type: 'boolean',
default: false,
demandOption: false
},
noCleanup: {
alias: 'nc',
describe: 'Do not delete the downloaded video file when an FFmpeg error occurs',
describe: 'Do not delete the downloaded video file when an FFmpeg error occurs.',
type: 'boolean',
default: false,
demandOption: false
Expand All @@ -97,13 +98,13 @@ export const argv: any = yargs.options({
demandOption: false
},
format: {
describe: 'Output container format (mkv, mp4, mov, anything that FFmpeg supports)',
describe: 'Output container format (mkv, mp4, mov, anything that FFmpeg supports).',
type: 'string',
default: 'mkv',
demandOption: false
},
skip: {
describe: 'Skip download if file already exists',
describe: 'Skip download if file already exists.',
type: 'boolean',
default: false,
demandOption: false
Expand Down
2 changes: 2 additions & 0 deletions src/destreamer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import cliProgress from 'cli-progress';
import fs from 'fs';
import isElevated from 'is-elevated';
import puppeteer from 'puppeteer';
import { ApiClient } from './ApiClient';


const { FFmpegCommand, FFmpegInput, FFmpegOutput } = require('@tedconf/fessonia')();
Expand Down Expand Up @@ -151,6 +152,7 @@ async function downloadVideo(videoGUIDs: Array<string>, outputDirectories: Array
if (argv.keepLoginCookies && index !== 0) {
logger.info('Trying to refresh token...');
session = await refreshSession('https://web.microsoftstream.com/video/' + videoGUIDs[index]);
ApiClient.getInstance().setSession(session);
}

const pbar: cliProgress.SingleBar = new cliProgress.SingleBar({
Expand Down