Skip to content

Commit

Permalink
HTTP: More granular tokenization (#2722)
Browse files Browse the repository at this point in the history
  • Loading branch information
@Jxck
Jxck committed Jan 18, 2021
1 parent 7a790bf commit 6183fd9
Show file tree
Hide file tree
Showing 4 changed files with 148 additions and 51 deletions.
45 changes: 35 additions & 10 deletions components/prism-http.js
View file
@@ -1,21 +1,46 @@
(function (Prism) {
Prism.languages.http = {
'request-line': {
pattern: /^(?:POST|GET|PUT|DELETE|OPTIONS|PATCH|TRACE|CONNECT)\s(?:https?:\/\/|\/)\S*\sHTTP\/[0-9.]+/m,
pattern: /^(?:GET|HEAD|POST|PUT|DELETE|CONNECT|OPTIONS|TRACE|PATCH|PRI|SEARCH)\s(?:https?:\/\/|\/)\S*\sHTTP\/[0-9.]+/m,

This comment has been minimized.

Sign in to view

Copy link
@yoshigev

yoshigev Feb 2, 2021

I'm not sure this is the place to ask this, but I want to know your opinion.
We are working on SIP, which has message syntax very similar to HTTP (see example message here).

For supporting it with PrismJS we could either add a new "language" for it, or make "http" also support it.
My question is: do you think this is important to really specify all the HTTP methods here or can we change it to something like [A-Z]+ (and also change the http-version to be [A-Z]+\/[0-9.])?
What do you say?

This comment has been minimized.

Sign in to view

Copy link
@RunDevelopment

RunDevelopment Feb 2, 2021

Member

HTTP only has to be able to highlight HTTP messages. Other message formats that are similar or supersets of HTTP are not supported, as is the case with (almost) every other language.

In the case of SIP, making a new language is the way to go. Since it's similar to HTTP, SIP can extend HTTP.

do you think this is important to really specify all the HTTP methods

Depends. You are right in that a simple pattern like [A-Z]+ will work for HTTP messages. However, we will then also highlight a lot of incorrect HTTP messages as if they were valid. But I don't see why we wouldn't want to match specifically these keywords. They are defined by HTTP's spec and there aren't too many of them. Again, HTTP only has to be able to highlight HTTP messages and nothing else.

This comment has been minimized.

Sign in to view

Copy link
@yoshigev

yoshigev Feb 2, 2021

Ok. Understood.
I'll try to find a way to extend http.

Thanks!
inside: {
// HTTP Verb
'property': /^(?:POST|GET|PUT|DELETE|OPTIONS|PATCH|TRACE|CONNECT)\b/,
// Path or query argument
'attr-name': /:\w+/
// HTTP Method
'method': {
pattern: /^[A-Z]+\b/,
alias: 'property'
},
// Request Target e.g. http://example.com, /path/to/file
'request-target': {
pattern: /^(\s)(?:https?:\/\/|\/)\S*(?=\s)/,
lookbehind: true,
alias: 'url'
},
// HTTP Version
'http-version': {
pattern: /^(\s)HTTP\/[0-9.]+/,
lookbehind: true,
alias: 'property'
},
}
},
'response-status': {
pattern: /^HTTP\/1.[01] \d.*/m,
pattern: /^HTTP\/[0-9.]+ \d+ .+/m,
inside: {
// Status, e.g. 200 OK
'property': {
pattern: /(^HTTP\/1.[01] )\d.*/i,
lookbehind: true
// HTTP Version
'http-version': {
pattern: /^HTTP\/[0-9.]+/,
alias: 'property'
},
// Status Code
'status-code': {
pattern: /^(\s)\d+(?=\s)/,
lookbehind: true,
alias: 'number'
},
// Reason Phrase
'reason-phrase': {
pattern: /^(\s).+/,
lookbehind: true,
alias: 'string'
}
}
},
Expand Down
2 changes: 1 addition & 1 deletion components/prism-http.min.js
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

130 changes: 99 additions & 31 deletions tests/languages/http/request-line_feature.test
View file
@@ -1,58 +1,126 @@
POST http://example.com HTTP/1.0
GET http://localhost:9999/foo.html HTTP/1.1
PUT http://www.example.com HTTP/2.0
DELETE https://example.com HTTP/1.1
OPTIONS https://www.example.com HTTP/1.1
PATCH http://example.com HTTP/1.0
TRACE http://example.com HTTP/1.0
CONNECT http://example.com HTTP/1.0
GET /path/to/foo.html HTTP/1.1
GET / HTTP/1.0
GET / HTTP/1.1
GET /path/to/file HTTP/1.1
GET /path/to/file?a=1&b=2 HTTP/1.1
GET http://example.com HTTP/1.1
GET https://example.com HTTP/1.1
GET https://example.com/ HTTP/1.1
GET https://example.com/path/to/file?a=1&b=2 HTTP/1.1
GET https://example.com:443/path/to/file?a=1&b=2 HTTP/1.1
GET https://user:pass@example.com:443/path/to/file?a=1&b=2 HTTP/1.1
HEAD / HTTP/1.1
POST / HTTP/1.1
PUT / HTTP/1.1
DELETE / HTTP/1.1
CONNECT / HTTP/1.1
OPTIONS / HTTP/1.1
TRACE / HTTP/1.1
PATCH / HTTP/1.1
PRI / HTTP/1.1
SEARCH / HTTP/1.1

----------------------------------------------------

[
["request-line", [
["property", "POST"],
" http://example.com HTTP/1.0"
["method", "GET"],
["request-target", "/"],
["http-version", "HTTP/1.0"]
]],
["request-line", [
["property", "GET"],
" http://localhost",
["attr-name", ":9999"],
"/foo.html HTTP/1.1"
["method", "GET"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["property", "PUT"],
" http://www.example.com HTTP/2.0"
["method", "GET"],
["request-target", "/path/to/file"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["property", "DELETE"],
" https://example.com HTTP/1.1"
["method", "GET"],
["request-target", "/path/to/file?a=1&b=2"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["property", "OPTIONS"],
" https://www.example.com HTTP/1.1"
["method", "GET"],
["request-target", "http://example.com"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["property", "PATCH"],
" http://example.com HTTP/1.0"
["method", "GET"],
["request-target", "https://example.com"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["property", "TRACE"],
" http://example.com HTTP/1.0"
["method", "GET"],
["request-target", "https://example.com/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["property", "CONNECT"],
" http://example.com HTTP/1.0"
["method", "GET"],
["request-target", "https://example.com/path/to/file?a=1&b=2"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["property", "GET"],
" /path/to/foo.html HTTP/1.1"
["method", "GET"],
["request-target", "https://example.com:443/path/to/file?a=1&b=2"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["property", "GET"],
" / HTTP/1.1"
["method", "GET"],
["request-target", "https://user:pass@example.com:443/path/to/file?a=1&b=2"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["method", "HEAD"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["method", "POST"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["method", "PUT"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["method", "DELETE"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["method", "CONNECT"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["method", "OPTIONS"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["method", "TRACE"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["method", "PATCH"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["method", "PRI"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]],
["request-line", [
["method", "SEARCH"],
["request-target", "/"],
["http-version", "HTTP/1.1"]
]]
]

Expand Down
22 changes: 13 additions & 9 deletions tests/languages/http/response-status_feature.test
View file
Expand Up @@ -7,23 +7,27 @@ HTTP/1.0 418 I'm a teapot

[
["response-status", [
"HTTP/1.0 ",
["property", "200 OK"]
["http-version", "HTTP/1.0"],
["status-code", "200"],
["reason-phrase", "OK"]
]],
["response-status", [
"HTTP/1.1 ",
["property", "403 Forbidden"]
["http-version", "HTTP/1.1"],
["status-code", "403"],
["reason-phrase", "Forbidden"]
]],
["response-status", [
"HTTP/1.1 ",
["property", "404 Not Found"]
["http-version", "HTTP/1.1"],
["status-code", "404"],
["reason-phrase", "Not Found"]
]],
["response-status", [
"HTTP/1.0 ",
["property", "418 I'm a teapot"]
["http-version", "HTTP/1.0"],
["status-code", "418"],
["reason-phrase", "I'm a teapot"]
]]
]

----------------------------------------------------

Checks for response statuses.
Checks for response statuses.

0 comments on commit 6183fd9

Please sign in to comment.