Scabot not building PRs recently

[retronym]

@SethTisue wonders:

Not sure why Jenkins isn't running

scala/scala#9656 (comment)

2021-08-04 18:08:55,006 [WARN] from scabot.server.Actors$PullRequestActor in application-akka.actor.default-dispatcher-53594 - FAILED launchBuild(scala-2.13.x-validate-main, scabot.core.BaseRef@57f193e8, 782888a, Map(repo_user -> scala, repo_name -> scala, _scabot_pr -> 9656, repo_ref -> 782888a)): spray.httpx.UnsuccessfulResponseException: Status: 422 Unprocessable Entity
Body: {"message":"Validation Failed","errors":[{"resource":"Status","code":"custom","field":"sha","message":"sha must be a 40 character SHA1"}],"documentation_url":"https://docs.github.com/rest/reference/repos#create-a-commit-status"}
2021-08-04 18:09:28,662 [WARN] from scabot.server.Actors$PullRequestActor in application-akka.actor.default-dispatcher-53594 - FAILED launchBuild(scala-2.13.x-validate-main, scabot.core.BaseRef@57f193e8, ddac496, Map(repo_user -> scala, repo_name -> scala, _scabot_pr -> 9656, repo_ref -> ddac496)): spray.httpx.UnsuccessfulResponseException: Status: 422 Unprocessable Entity
Body: {"message":"Validation Failed","errors":[{"resource":"Status","code":"custom","field":"sha","message":"sha must be a 40 character SHA1"}],"documentation_url":"https://docs.github.com/rest/reference/repos#create-a-commit-status"}

[retronym]

It is possible that this error was just due to insufficient input validation on the /rebuild command, if this is issues with a non-40 character SHA.

The general problem with builds not happening could be something else. I'll try bouncing it.

[retronym]

I'm trying to increase the logging. I think there is another problem -- the push-to-deploy might be broken because of the master -> main branch rename.

[retronym]

Found the logs with:

 journalctl -u scabot -b -f

Aug 05 06:01:29 ip-172-31-10-237 run[13935]: DEBUG s.c.c.HttpClientConnection - Attempting connection to scala-ci.typesafe.com/54.67.111.226:443
Aug 05 06:01:29 ip-172-31-10-237 run[13935]: DEBUG a.i.TcpOutgoingConnection - Attempting connection to [scala-ci.typesafe.com/54.67.111.226:443]
Aug 05 06:01:29 ip-172-31-10-237 run[13935]: DEBUG a.i.TcpOutgoingConnection - Connection established to [scala-ci.typesafe.com/54.67.111.226:443]
Aug 05 06:01:29 ip-172-31-10-237 run[13935]: DEBUG s.c.c.HttpClientConnection - Connected to scala-ci.typesafe.com/54.67.111.226:443
Aug 05 06:01:29 ip-172-31-10-237 run[13935]: DEBUG s.c.c.HttpHostConnectionSlot - Connection to scala-ci.typesafe.com:443 established, dispatching 1 pending requests
Aug 05 06:01:29 ip-172-31-10-237 run[13935]: DEBUG s.c.c.HttpHostConnectionSlot - Dispatching POST request to /job/scala-2.13.x-validate-main/buildWithParameters?repo_name=scala&_scabot_pr=9656&repo_ref=d200f2a1e95df072c77f1707a27eb6f70e7bd10c&repo_user=scala&_scabot_last=1 across connection Actor[akka://application/user/IO-HTTP/group-0/12#-1735849917]
Aug 05 06:01:29 ip-172-31-10-237 run[13935]: DEBUG s.c.c.HttpHostConnectionSlot - Delivering 201 Created response for POST request to /repos/scala/scala/statuses/d200f2a1e95df072c77f1707a27eb6f70e7bd10c
Aug 05 06:01:29 ip-172-31-10-237 run[13935]: DEBUG a.a.ActorSystemImpl - {"url":"https://api.github.com/repos/scala/scala/statuses/d200f2a1e95df072c77f1707a27eb6f70e7bd10c","avatar_url":"https://avatars.githubusercontent.com/u/1563693?v=4","id":14006928690,"node_id":"MDEzOlN0YXR1c0NvbnRleHQxNDAwNjkyODY5MA==","state":"pending","description":"Found earlier commit(s) marked pending: 782888, ddac49","target_url":null,"context":"combined","created_at":"2021-08-05T06:01:29Z","updated_at":"2021-08-05T06:01:29Z","creator":{"login":"scala-jenkins","id":1563693,"node_id":"MDQ6VXNlcjE1NjM2OTM=","avatar_url":"https://avatars.githubusercontent.com/u/1563693?v=4","gravatar_id":"","url":"https://api.github.com/users/scala-jenkins","html_url":"https://github.com/scala-jenkins","followers_url":"https://api.github.com/users/scala-jenkins/followers","following_url":"https://api.github.com/users/scala-jenkins/following{/other_user}","gists_url":"https://api.github.com/users/scala-jenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/scala-jenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scala-jenkins/subscriptions","organizations_url":"https://api.github.com/users/scala-jenkins/orgs","repos_url":"https://api.github.com/users/scala-jenkins/repos","events_url":"https://api.github.com/users/scala-jenkins/events{/privacy}","received_events_url":"https://api.github.com/users/scala-jenkins/received_events","type":"User","site_admin":false}}
Aug 05 06:01:30 ip-172-31-10-237 run[13935]: DEBUG s.c.c.HttpHostConnectionSlot - Delivering 201 Created response for POST request to /job/scala-2.13.x-validate-main/buildWithParameters?repo_name=scala&_scabot_pr=9656&repo_ref=d200f2a1e95df072c77f1707a27eb6f70e7bd10c&repo_user=scala&_scabot_last=1
Aug 05 06:01:30 ip-172-31-10-237 run[13935]: DEBUG a.a.ActorSystemImpl -
Aug 05 06:01:30 ip-172-31-10-237 run[13935]: INFO s.s.Actors$PullRequestActor - Launched scala-2.13.x-validate-main for d200f2a1e95df072c77f1707a27eb6f70e7bd10c:
Aug 05 06:01:58 ip-172-31-10-237 run[13935]: DEBUG s.c.c.HttpClientConnection - Dropping Close since the SSL connection is already closing
Aug 05 06:01:58 ip-172-31-10-237 run[13935]: DEBUG s.c.c.HttpClientConnection - Connection was PeerClosed, awaiting TcpConnection termination...
Aug 05 06:01:58 ip-172-31-10-237 run[13935]: DEBUG s.c.c.HttpClientConnection - TcpConnection terminated, stopping
Aug 05 06:01:59 ip-172-31-10-237 run[13935]: DEBUG o.a.h.i.c.PoolingClientConnectionManager - Closing connections idle longer than 60 SECONDS
Aug 05 06:02:28 ip-172-31-10-237 run[13935]: DEBUG s.c.c.HttpHostConnectionSlot - Initiating idle shutdown

But nothing at all showing up on Jenkins:

[retronym]

Oh.

[retronym]

I manually deleted /var/opt/jfrog/artifactory/log/archived/*.log to free up enough space to bring jenkins master back online. Maybe the recent artifactory upgrade increases the amount or retained rolled logs?

[retronym]

Updated the docs about service restarts and how to read the log files in #111