Remove mathoids PNG images

[physikerwelt]

Remove mathoids PNG images from RestBase endpoint.

• Stop storing PNG response from mathoid
• Redirect PNG endpoint to SVG endpoint

Unfortunately, one can not remove the PNG option from swagger, as the same .yaml file is used for the swagger ui and the parameter check.

See https://gerrit.wikimedia.org/r/c/mediawiki/services/mathoid/+/875405/8/test/restbase/README.md for local testing.

Bug: T334842
Change-Id: I6e16ccb5f3d84e76393ae72172a91e77a8c8b447

[physikerwelt]

It's a bit hard to test.

1. Use REL1_39 of core and extensions (I could not get REL1_38 to run with my setup)
2. Use REL_38 of math extension
3. Set $wgMathFullRestbaseURL='http://localhost:7231/localhost/'; in LocalSettings.php
4. Set up restbase as described here https://gerrit.wikimedia.org/r/c/mediawiki/services/mathoid/+/875405/8/test/restbase/README.md
5. checkout this patch
6. Add a test formula E=mc^2
7. Log in and switch rendering mode PNG
8. Check if you see SVG image regardless.

[physikerwelt]

@brightbyte @Ladsgroup what do you think?

[Ladsgroup]

I'm all for it. I can go ask Virginia if needed (today is public holiday in US though)

[brightbyte]

I think it's good to go, and I agree we should do it. Virginia told me she is fine with it, but it would be nice if she could comment here.

I don't have capacity to manually test this, and I don't know the code well enough to just merge after eyeballing it... We'll have to find someone (tm) who can do one or the other. Maybe @johngian could get this deployed to beta first, so we can try it there?

[physikerwelt]

@brightbyte @Ladsgroup thank you both. I think we definitely need some testing.
Deploying to beta still requires installing REL1_38, to see if the images look good in the browser.

As an alternative, I can set up a demo in beta.math.wmflabs.org. I did already updated Mathoid on https://mathoid.beta.math.wmflabs.org to the version of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/919375 and built a custom Mediawiki container with REL1_38. However, I have difficulties finding a docker image of restbase. I was searching here at https://docker-registry.wikimedia.org/ but I need clarification on the current version.

Is there already a base image, I can use to build a demo of this PR, or should I start with a blank node container?

[brightbyte]

I'd be fine with being able to just test Mathoid by hand for now.

Do we know public MediaWiki instances that are still asking for png?

[johngian]

We don't have a dockerized version of restbase published. The service is running on servers with system wide installed node runtime. For the wikifeeds upgrade to node 16 I used this as a base image: https://docker-registry.wikimedia.org/nodejs16-slim. If you want to be closer to current production you need to use node 10. There are a few base images already published on wmf docker registry.

[johngian]

Also for convenience we can always point beta restbase to the new mathoid instance so we don't need to put more effort on setting up restbase. I wouldn't consider it a huge deal in case it causes problem just for beta wikipedia.

[physikerwelt]

I don't expect problems for beta wikipedia. The problem is how browsers will handle svgs being announced as PNG. Current firefox works well, but...

[physikerwelt]

I set up a test environment on the labs instance math19. You can log into the instance if you have labs account. The docker config is in /home/physikerwelt/srv-math19. While setup, I realized that the table mathoid_png needs to exist, so I added one additional commit.
I hope this helps with testing.
If you want, I can also set up a demo 1_38 instance, which is a bit more effort than expected as the database schema changed in between, and I would also need to set up additional database config.

See below how you can test locally (the webui did never work). You need to look for x-resource-location in the check response to receive the link to the image or mathml output.

physikerwelt@a1 /tmp % curl -vd 'q=E=mc^2'  https://restbase.beta.math.wmflabs.org/localhost/v1/media/math/check/tex
*   Trying 185.15.56.41:443...
* Connected to restbase.beta.math.wmflabs.org (185.15.56.41) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=restbase.beta.math.wmflabs.org
*  start date: Jun 21 07:12:20 2023 GMT
*  expire date: Sep 19 07:12:19 2023 GMT
*  subjectAltName: host "restbase.beta.math.wmflabs.org" matched cert's "restbase.beta.math.wmflabs.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: POST]
* h2h3 [:path: /localhost/v1/media/math/check/tex]
* h2h3 [:scheme: https]
* h2h3 [:authority: restbase.beta.math.wmflabs.org]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* h2h3 [content-length: 8]
* h2h3 [content-type: application/x-www-form-urlencoded]
* Using Stream ID: 1 (easy handle 0x7fb0d1010a00)
> POST /localhost/v1/media/math/check/tex HTTP/2
> Host: restbase.beta.math.wmflabs.org
> user-agent: curl/7.88.1
> accept: */*
> content-length: 8
> content-type: application/x-www-form-urlencoded
> 
* We are completely uploaded and fine
< HTTP/2 200 
< access-control-allow-headers: accept, content-type, content-length, cache-control, accept-language, api-user-agent, if-match, if-modified-since, if-none-match, dnt, accept-encoding
< access-control-allow-methods: GET,HEAD
< access-control-allow-origin: *
< access-control-expose-headers: etag
< cache-control: no-cache
< content-security-policy: default-src 'none'; frame-ancestors 'none'
< content-type: application/json
< date: Wed, 21 Jun 2023 09:08:43 GMT
< referrer-policy: origin-when-cross-origin
< server: f8e557cd703b
< x-content-security-policy: default-src 'none'; frame-ancestors 'none'
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< x-request-id: 38571c00-1013-11ee-ae5a-4dd5d3c8fa76
< x-resource-location: 4c0004393a88f350a93bcef62106d556c7fc827b
< x-webkit-csp: default-src 'none'; frame-ancestors 'none'
< x-xss-protection: 1; mode=block
< 
* Connection #0 to host restbase.beta.math.wmflabs.org left intact
{"success":true,"checked":"E=mc^{2}","requiredPackages":[],"identifiers":["E","m","c"],"endsWithDot":false}%                                                                                                                                                  physikerwelt@a1 /tmp % curl -X 'GET' \
  'https://restbase.beta.math.wmflabs.org/localhost/v1/media/math/render/png/4c0004393a88f350a93bcef62106d556c7fc827b' \
  -H 'accept: image/svg+xml'
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="8.976ex" height="2.676ex" style="vertical-align: -0.338ex;" viewBox="0 -1006.6 3864.5 1152.1" role="img" focusable="false" xmlns="http://www.w3.org/2000/svg" aria-labelledby="MathJax-SVG-1-Title"><title id="MathJax-SVG-1-Title">upper E equals m c squared</title><defs aria-hidden="true"><path stroke-width="1" id="E1-MJMATHI-45" d="M492 213Q472 213 472 226Q472 230 477 250T482 285Q482 316 461 323T364 330H312Q311 328 277 192T243 52Q243 48 254 48T334 46Q428 46 458 48T518 61Q567 77 599 117T670 248Q680 270 683 272Q690 274 698 274Q718 274 718 261Q613 7 608 2Q605 0 322 0H133Q31 0 31 11Q31 13 34 25Q38 41 42 43T65 46Q92 46 125 49Q139 52 144 61Q146 66 215 342T285 622Q285 629 281 629Q273 632 228 634H197Q191 640 191 642T193 659Q197 676 203 680H757Q764 676 764 669Q764 664 751 557T737 447Q735 440 717 440H705Q698 445 698 453L701 476Q704 500 704 528Q704 558 697 578T678 609T643 625T596 632T532 634H485Q397 633 392 631Q388 629 386 622Q385 619 355 499T324 377Q347 376 372 376H398Q464 376 489 391T534 472Q538 488 540 490T557 493Q562 493 565 493T570 492T572 491T574 487T577 483L544 351Q511 218 508 216Q505 213 492 213Z"></path><path stroke-width="1" id="E1-MJMAIN-3D" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"></path><path stroke-width="1" id="E1-MJMATHI-6D" d="M21 287Q22 293 24 303T36 341T56 388T88 425T132 442T175 435T205 417T221 395T229 376L231 369Q231 367 232 367L243 378Q303 442 384 442Q401 442 415 440T441 433T460 423T475 411T485 398T493 385T497 373T500 364T502 357L510 367Q573 442 659 442Q713 442 746 415T780 336Q780 285 742 178T704 50Q705 36 709 31T724 26Q752 26 776 56T815 138Q818 149 821 151T837 153Q857 153 857 145Q857 144 853 130Q845 101 831 73T785 17T716 -10Q669 -10 648 17T627 73Q627 92 663 193T700 345Q700 404 656 404H651Q565 404 506 303L499 291L466 157Q433 26 428 16Q415 -11 385 -11Q372 -11 364 -4T353 8T350 18Q350 29 384 161L420 307Q423 322 423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 181Q151 335 151 342Q154 357 154 369Q154 405 129 405Q107 405 92 377T69 316T57 280Q55 278 41 278H27Q21 284 21 287Z"></path><path stroke-width="1" id="E1-MJMATHI-63" d="M34 159Q34 268 120 355T306 442Q362 442 394 418T427 355Q427 326 408 306T360 285Q341 285 330 295T319 325T330 359T352 380T366 386H367Q367 388 361 392T340 400T306 404Q276 404 249 390Q228 381 206 359Q162 315 142 235T121 119Q121 73 147 50Q169 26 205 26H209Q321 26 394 111Q403 121 406 121Q410 121 419 112T429 98T420 83T391 55T346 25T282 0T202 -11Q127 -11 81 37T34 159Z"></path><path stroke-width="1" id="E1-MJMAIN-32" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"></path></defs><g stroke="currentColor" fill="currentColor" stroke-width="0" transform="matrix(1 0 0 -1 0 0)" aria-hidden="true"> <use xlink:href="#E1-MJMATHI-45" x="0" y="0"></use> <use xlink:href="#E1-MJMAIN-3D" x="1042" y="0"></use> <use xlink:href="#E1-MJMATHI-6D" x="2098" y="0"></use><g transform="translate(2977,0)"> <use xlink:href="#E1-MJMATHI-63" x="0" y="0"></use> <use transform="scale(0.707)" xlink:href="#E1-MJMAIN-32" x="613" y="583"></use></g></g></svg>% 

[brightbyte]

Where do I find the test instance? https://math19.wmflabs.org/ and https://beta.math19.wmflabs.org/ both don't work.

[physikerwelt]

Where do I find the test instance? https://math19.wmflabs.org/ and https://beta.math19.wmflabs.org/ both don't work.

https://restbase.beta.math.wmflabs.org/localhost/v1/

[brightbyte]

Ok, I confirmed that this is working as follows:

curl -v -X 'POST' 'https://restbase.beta.math.wmflabs.org/localhost/v1/media/math/check/inline-tex' -H 'accept: application/json' -H 'Content-Type: application/json' --data '{ "q": "x_1" }'

I received {"success":true,"checked":"x_{1}","requiredPackages":[],"identifiers":["x_{1}"],"endsWithDot":false} and x-resource-location: 9221bc8e597b2308489fe8e8e128f8045f4e7a0e.

from this response, I constructed the URL https://restbase.beta.math.wmflabs.org/localhost/v1/media/math/render/png/9221bc8e597b2308489fe8e8e128f8045f4e7a0e which indeed returns an SVG with content-type: image/svg+xml; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/SVG/1.0.0", which Firefox happily displays.

[brightbyte]

@johngian Looks good to me, do you think this can go in?

[physikerwelt]

Has this also been deployed?

[johngian]

I just deployed this change